Uploaded image for project: 'ROOT'
  1. ROOT
  2. ROOT-9783 RDataFrame-related tickets
  3. ROOT-9133

[DF] Large memory usage when writing uncompressed root files from multiple threads

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Closed (View Workflow)
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 6.12/00
    • Fix Version/s: 6.18/00
    • Component/s: Other, TTree
    • Labels:
      None
    • Environment:

      any

      Description

      It seems there is a memory usage issue when creating an uncompressed ROOT file that contains a TTree in a multi-thread application.

      Here is a short TDataFrame snippet that writes 4 million events from multiple threads. Events contain one single branch of type std::vector<double> with 100 values (0 to 99).

      #include <ROOT/TDataFrame.hxx>
       
      int main()
      {
         ROOT::EnableImplicitMT();
         ROOT::Experimental::TDataFrame d(4000000);
         auto gencol = []() {
            std::vector<double> v;
            v.reserve(100);
            for (int i = 0; i < 100; ++i)
               v.emplace_back(i);
            return v;
         };
         ROOT::Experimental::TDF::TSnapshotOptions opts;
         opts.fCompressionLevel = 0;
         d.Define("col0", gencol).Snapshot<std::vector<double>>("t", "ofile.root", {"col0"}, opts);
         return 0;
      }

      Multi-thread (8 threads) and uncompressed file:
      32 seconds, ~4GB max RAM usage

      Single-thread and uncompressed file:
      22 seconds, 200MB max RAM usage

      Multi-thread (8 threads) and compressed file:
      4 seconds, 400MB max RAM usage

      Single-thread and compressed file:
      10 seconds, 250MB max RAM usage

      I was able to somehow replicate the issue without TDataFrame. Here is the snippet:

      #include <ROOT/TBufferMerger.hxx>
      #include <TFile.h>
      #include <TROOT.h>
      #include <TTree.h>
       
      static void Fill(TTree *tree, int count)
      {
         std::vector<double> col0;
         tree->Branch("col0", &col0);
         for (int i = 0; i < count; ++i) {
            col0 = []() {
               std::vector<double> v;
               v.reserve(100);
               for (int i = 0; i < 100; ++i)
                  v.emplace_back(i);
               return v;
            }();
            tree->Fill();
         }
         tree->ResetBranchAddresses();
      }
       
      int main()
      {
         constexpr int nThreads = 8;
         constexpr int evtsPerThread = 4000000 / 8;
       
         ROOT::EnableImplicitMT();
       
         ROOT::Experimental::TBufferMerger merger("testwrite.root", "RECREATE", /*compress=*/0);
         std::vector<std::thread> threads;
         for (int i = 0; i < nThreads; ++i) {
            threads.emplace_back([&merger]() {
               auto myfile = merger.GetFile();
               auto mytree = new TTree("t", "t");
               mytree->ResetBit(kMustCleanup);
               Fill(mytree, evtsPerThread);
               myfile->Write();
            });
         }
       
         for (auto &t : threads)
            t.join();
         return 0;
      }

      Multi-thread (8 threads) and uncompressed file:
      26 seconds, ~8GB max RAM usage

      Multi-thread (8 threads) and compressed file:
      4 seconds, 570MB max RAM usage

      Single-thread execution does not make sense with the second snippet, but I tried calling EnableThreadSafety instead of EnableImplicitMT, so that TTree::Fill is not executed in IMT mode.

      noIMT, 8 threads, uncompressed file:
      36 seconds, ~9GB max RAM usage

      noIMT, 8 threads, compressed file:
      4 seconds, 490MB max RAM usage

        Attachments

        1. memusage-after.png
          memusage-after.png
          19 kB
        2. memusage-before.png
          memusage-before.png
          8 kB
        3. root-9133.svg
          271 kB

          Issue Links

            Activity

              People

              • Assignee:
                amadio Guilherme Amadio
                Reporter:
                eguiraud Enrico Guiraud
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  PlannedEnd:
                  PlannedStart:
                  Actual Start:
                  Actual End: