Uploaded image for project: 'ROOT'
  1. ROOT
  2. ROOT-5616

hadd problem: "double free or corruption" on multiple usage

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 5.30/00, 5.34/00
    • Fix Version/s: None
    • Component/s: Other
    • Labels:
      None
    • Environment:

      Ubuntu 13.10

      Description

      Hi,

      When I use hadd multiple times (for example because I have a lot of files to merge) it results somehow in a corruption of data.

      I attached as a minimal example the 4 files a1 - a4.root. If I hadd them all in one step nothing happens (at least not until I try to further process them).
      But if I do it like this:
      hadd b1.root a1.root a2.root
      hadd b2.root a3.root a4.root
      hadd c1.root b1.root b2.root
      it crashes like this (including gdb):

      robin@robin-laptop:~/rootfiles/test$ gdb hadd
      GNU gdb (GDB) 7.6.1-ubuntu
      Copyright (C) 2013 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/root/root_v5.34.10/bin/hadd...(no debugging symbols found)...done.
      (gdb) run c1.root b1.root b2.root
      Starting program: /opt/root/root_v5.34.10/bin/hadd c1.root b1.root b2.root
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      hadd Target file: c1.root
      hadd Source file 1: b1.root
      hadd Source file 2: b2.root
      hadd Target path: c1.root:/

          • Error in `/opt/root/root_v5.34.10/bin/hadd': double free or corruption (!prev): 0x0000000001003ea0 ***
            ======= Backtrace: =========
            /lib/x86_64-linux-gnu/libc.so.6(+0x80996)[0x7ffff667e996]
            /opt/root/root_v5.34.10/lib/libCore.so(_ZN8TStorage11ReAllocCharEPcmm+0x79)[0x7ffff70adcf9]
            /opt/root/root_v5.34.10/lib/libCore.so(_ZN7TBuffer6ExpandEib+0x46)[0x7ffff70736b6]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN7TBasket17LoadBasketBuffersExiP5TFileP5TTree+0x201)[0x7ffff4133fa1]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN11TTreeCloner12WriteBasketsEv+0x68)[0x7ffff417f438]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN11TTreeCloner4ExecEv+0x45)[0x7ffff417fba5]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN5TTree11CopyEntriesEPS_xPKc+0x484)[0x7ffff4190c74]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN5TTree9CloneTreeExPKc+0x41f)[0x7ffff418cd3f]
            /opt/root/root_v5.34.10/lib/libTree.so(_ZN5TTree5MergeEP11TCollectionP14TFileMergeInfo+0x181)[0x7ffff4187b91]
            /opt/root/root_v5.34.10/lib/libTree.so(+0x1059d6)[0x7ffff41ad9d6]
            /opt/root/root_v5.34.10/lib/libRIO.so(_ZN11TFileMerger14MergeRecursiveEP10TDirectoryP5TListi+0xb44)[0x7ffff799c854]
            /opt/root/root_v5.34.10/lib/libRIO.so(_ZN11TFileMerger12PartialMergeEi+0xdd)[0x7ffff799d98d]
            /opt/root/root_v5.34.10/bin/hadd(main+0x6c3)[0x401ea3]
            /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff661fde5]
            /opt/root/root_v5.34.10/bin/hadd[0x402578]
            ======= Memory map: ========
            00400000-00404000 r-xp 00000000 08:03 1323658 /opt/root/root_v5.34.10/bin/hadd
            00603000-00604000 r--p 00003000 08:03 1323658 /opt/root/root_v5.34.10/bin/hadd
            00604000-00605000 rw-p 00004000 08:03 1323658 /opt/root/root_v5.34.10/bin/hadd
            00605000-0102f000 rw-p 00000000 00:00 0 [heap]
            7ffff1bf9000-7ffff1c25000 r-xp 00000000 08:03 1447253 /opt/root/root_v5.34.10/lib/libCintex.so
            7ffff1c25000-7ffff1e24000 ---p 0002c000 08:03 1447253 /opt/root/root_v5.34.10/lib/libCintex.so
            7ffff1e24000-7ffff1e25000 r--p 0002b000 08:03 1447253 /opt/root/root_v5.34.10/lib/libCintex.so
            7ffff1e25000-7ffff1e26000 rw-p 0002c000 08:03 1447253 /opt/root/root_v5.34.10/lib/libCintex.so
            7ffff1e26000-7ffff1ec0000 r-xp 00000000 08:03 1446819 /opt/root/root_v5.34.10/lib/libReflex.so
            7ffff1ec0000-7ffff20bf000 ---p 0009a000 08:03 1446819 /opt/root/root_v5.34.10/lib/libReflex.so
            7ffff20bf000-7ffff20c3000 r--p 00099000 08:03 1446819 /opt/root/root_v5.34.10/lib/libReflex.so
            7ffff20c3000-7ffff20c5000 rw-p 0009d000 08:03 1446819 /opt/root/root_v5.34.10/lib/libReflex.so
            7ffff20c5000-7ffff230d000 r-xp 00000000 08:03 1446900 /opt/root/root_v5.34.10/lib/libMatrix.so
            7ffff230d000-7ffff250c000 ---p 00248000 08:03 1446900 /opt/root/root_v5.34.10/lib/libMatrix.so
            7ffff250c000-7ffff2515000 r--p 00247000 08:03 1446900 /opt/root/root_v5.34.10/lib/libMatrix.so
            7ffff2515000-7ffff251b000 rw-p 00250000 08:03 1446900 /opt/root/root_v5.34.10/lib/libMatrix.so
            7ffff251b000-7ffff251f000 rw-p 00000000 00:00 0
            7ffff251f000-7ffff25bb000 r-xp 00000000 08:03 6954824 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.10.1
            7ffff25bb000-7ffff27bb000 ---p 0009c000 08:03 6954824 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.10.1
            7ffff27bb000-7ffff27c0000 r--p 0009c000 08:03 6954824 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.10.1
            7ffff27c0000-7ffff27c1000 rw-p 000a1000 08:03 6954824 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.10.1
            7ffff27c1000-7ffff2908000 r-xp 00000000 08:03 1446899 /opt/root/root_v5.34.10/lib/libNet.so
            7ffff2908000-7ffff2b08000 ---p 00147000 08:03 1446899 /opt/root/root_v5.34.10/lib/libNet.so
            7ffff2b08000-7ffff2b14000 r--p 00147000 08:03 1446899 /opt/root/root_v5.34.10/lib/libNet.so
            7ffff2b14000-7ffff2b18000 rw-p 00153000 08:03 1446899 /opt/root/root_v5.34.10/lib/libNet.so
            7ffff2b18000-7ffff2b1b000 rw-p 00000000 00:00 0
            7ffff2b1b000-7ffff2d40000 r-xp 00000000 08:03 1446834 /opt/root/root_v5.34.10/lib/libMathCore.so
            7ffff2d40000-7ffff2f40000 ---p 00225000 08:03 1446834 /opt/root/root_v5.34.10/lib/libMathCore.so
            7ffff2f40000-7ffff2f49000 r--p 00225000 08:03 1446834 /opt/root/root_v5.34.10/lib/libMathCore.so
            7ffff2f49000-7ffff2f4d000 rw-p 0022e000 08:03 1446834 /opt/root/root_v5.34.10/lib/libMathCore.so
            7ffff2f4d000-7ffff2f54000 rw-p 00000000 00:00 0
            7ffff2f54000-7ffff303b000 r-xp 00000000 08:03 1447004 /opt/root/root_v5.34.10/lib/libGpad.so
            7ffff303b000-7ffff323b000 ---p 000e7000 08:03 1447004 /opt/root/root_v5.34.10/lib/libGpad.so
            7ffff323b000-7ffff3244000 r--p 000e7000 08:03 1447004 /opt/root/root_v5.34.10/lib/libGpad.so
            7ffff3244000-7ffff3247000 rw-p 000f0000 08:03 1447004 /opt/root/root_v5.34.10/lib/libGpad.so
            7ffff3247000-7ffff3249000 rw-p 00000000 00:00 0
            7ffff3249000-7ffff373c000 r-xp 00000000 08:03 1446902 /opt/root/root_v5.34.10/lib/libHist.so
            7ffff373c000-7ffff393b000 ---p 004f3000 08:03 1446902 /opt/root/root_v5.34.10/lib/libHist.so
            7ffff393b000-7ffff3963000 r--p 004f2000 08:03 1446902 /opt/root/root_v5.34.10/lib/libHist.so
            7ffff3963000-7ffff396a000 rw-p 0051a000 08:03 1446902 /opt/root/root_v5.34.10/lib/libHist.so
            7ffff396a000-7ffff3974000 rw-p 00000000 00:00 0
            7ffff3974000-7ffff3b5e000 r-xp 00000000 08:03 1447002 /opt/root/root_v5.34.10/lib/libGraf.so
            7ffff3b5e000-7ffff3d5d000 ---p 001ea000 08:03 1447002 /opt/root/root_v5.34.10/lib/libGraf.so
            7ffff3d5d000-7ffff3d6a000 r--p 001e9000 08:03 1447002 /opt/root/root_v5.34.10/lib/libGraf.so
            7ffff3d6a000-7ffff3d70000 rw-p 001f6000 08:03 1447002 /opt/root/root_v5.34.10/lib/libGraf.so
            7ffff3d70000-7ffff3d7e000 rw-p 00000000 00:00 0
            7ffff3d7e000-7ffff3e97000 r-xp 00000000 08:03 1447005 /opt/root/root_v5.34.10/lib/libGraf3d.so
            7ffff3e97000-7ffff4097000 ---p 00119000 08:03 1447005 /opt/root/root_v5.34.10/lib/libGraf3d.so
            7ffff4097000-7ffff40a1000 r--p 00119000 08:03 1447005 /opt/root/root_v5.34.10/lib/libGraf3d.so
            7ffff40a1000-7ffff40a4000 rw-p 00123000 08:03 1447005 /opt/root/root_v5.34.10/lib/libGraf3d.so
            7ffff40a4000-7ffff40a8000 rw-p 00000000 00:00 0
            7ffff40a8000-7ffff42ce000 r-xp 00000000 08:03 1447001 /opt/root/root_v5.34.10/lib/libTree.so
            7ffff42ce000-7ffff44ce000 ---p 00226000 08:03 1447001 /opt/root/root_v5.34.10/lib/libTree.so
            7ffff44ce000-7ffff44dd000 r--p 00226000 08:03 1447001 /opt/root/root_v5.34.10/lib/libTree.so
            7ffff44dd000-7ffff44e2000 rw-p 00235000 08:03 1447001 /opt/root/root_v5.34.10/lib/libTree.so
            7ffff44e2000-7ffff44e6000 rw-p 00000000 00:00 0
            7ffff44e6000-7ffff469a000 r-xp 00000000 08:03 1447057 /opt/root/root_v5.34.10/lib/libTreePlayer.so
            7ffff469a000-7ffff489a000 ---p 001b4000 08:03 1447057 /opt/root/root_v5.34.10/lib/libTreePlayer.so
            7ffff489a000-7ffff48a3000 r--p 001b4000 08:03 1447057 /opt/root/root_v5.34.10/lib/libTreePlayer.so
            7ffff48a3000-7ffff48a6000 rw-p 001bd000 08:03 1447057 /opt/root/root_v5.34.10/lib/libTreePlayer.so
            7ffff48a6000-7ffff48ac000 rw-p 00000000 00:00 0
            7ffff48ac000-7ffff48b8000 r-xp 00000000 08:03 921478 /lib/x86_64-linux-gnu/libnss_files-2.17.so
            7ffff48b8000-7ffff4ab7000 ---p 0000c000 08:03 921478 /lib/x86_64-linux-gnu/libnss_files-2.17.so
            7ffff4ab7000-7ffff4ab8000 r--p 0000b000 08:03 921478 /lib/x86_64-linux-gnu/libnss_files-2.17.so
            7ffff4ab8000-7ffff4ab9000 rw-p 0000c000 08:03 921478 /lib/x86_64-linux-gnu/libnss_files-2.17.so
            7ffff4ab9000-7ffff4ac4000 r-xp 00000000 08:03 921482 /lib/x86_64-linux-gnu/libnss_nis-2.17.so
            7ffff4ac4000-7ffff4cc3000 ---p 0000b000 08:03 921482 /lib/x86_64-linux-gnu/libnss_nis-2.17.so
            7ffff4cc3000-7ffff4cc4000 r--p 0000a000 08:03 921482 /lib/x86_64-linux-gnu/libnss_nis-2.17.so
            7ffff4cc4000-7ffff4cc5000 rw-p 0000b000 08:03 921482 /lib/x86_64-linux-gnu/libnss_nis-2.17.so
            7ffff4cc5000-7ffff4cdc000 r-xp 00000000 08:03 921472 /lib/x86_64-linux-gnu/libnsl-2.17.so
            7ffff4cdc000-7ffff4edb000 ---p 00017000 08:03 921472 /lib/x86_64-linux-gnu/libnsl-2.17.so
            7ffff4edb000-7ffff4edc000 r--p 00016000 08:03 921472 /lib/x86_64-linux-gnu/libnsl-2.17.so
            7ffff4edc000-7ffff4edd000 rw-p 00017000 08:03 921472 /lib/x86_64-linux-gnu/libnsl-2.17.so
            7ffff4edd000-7ffff4edf000 rw-p 00000000 00:00 0
            7ffff4edf000-7ffff4ee7000 r-xp 00000000 08:03 921474 /lib/x86_64-linux-gnu/libnss_compat-2.17.so
            7ffff4ee7000-7ffff50e6000 ---p 00008000 08:03 921474 /lib/x86_64-linux-gnu/libnss_compat-2.17.so
            7ffff50e6000-7ffff50e7000 r--p 00007000 08:03 921474 /lib/x86_64-linux-gnu/libnss_compat-2.17.so
            7ffff50e7000-7ffff50e8000 rw-p 00008000 08:03 921474 /lib/x86_64-linux-gnu/libnss_compat-2.17.so
            7ffff50e8000-7ffff50ff000 r-xp 00000000 08:03 921517 /lib/x86_64-linux-gnu/libpthread-2.17.so
            7ffff50ff000-7ffff52ff000 ---p 00017000 08:03 921517 /lib/x86_64-linux-gnu/libpthread-2.17.so
            7ffff52ff000-7ffff5300000 r--p 00017000 08:03 921517 /lib/x86_64-linux-gnu/libpthread-2.17.so
            7ffff5300000-7ffff5301000 rw-p 00018000 08:03 921517 /lib/x86_64-linux-gnu/libpthread-2.17.so
            7ffff5301000-7ffff5305000 rw-p 00000000 00:00 0
            7ffff5305000-7ffff531d000 r-xp 00000000 08:03 921560 /lib/x86_64-linux-gnu/libz.so.1.2.8
            7ffff531d000-7ffff551c000 ---p 00018000 08:03 921560 /lib/x86_64-linux-gnu/libz.so.1.2.8
            Program received signal SIGABRT, Aborted.
            0x00007ffff6634f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
            56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
            (gdb)
            (gdb) bt
            #0 0x00007ffff6634f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
            #1 0x00007ffff66385e8 in __GI_abort () at abort.c:90
            #2 0x00007ffff66724fb in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6786240 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:199
            #3 0x00007ffff667e996 in malloc_printerr (ptr=0x1003ea0, str=0x7ffff6786328 "double free or corruption (!prev)", action=3) at malloc.c:4923
            #4 _int_free (av=<optimized out>, p=0x1003e90, have_lock=0) at malloc.c:3779
            #5 0x00007ffff70adcf9 in TStorage::ReAllocChar(char*, unsigned long, unsigned long) () from /opt/root/root_v5.34.10/lib/libCore.so
            #6 0x00007ffff70736b6 in TBuffer::Expand(int, bool) () from /opt/root/root_v5.34.10/lib/libCore.so
            #7 0x00007ffff4133fa1 in TBasket::LoadBasketBuffers(long long, int, TFile*, TTree*) () from /opt/root/root_v5.34.10/lib/libTree.so
            #8 0x00007ffff417f438 in TTreeCloner::WriteBaskets() () from /opt/root/root_v5.34.10/lib/libTree.so
            #9 0x00007ffff417fba5 in TTreeCloner::Exec() () from /opt/root/root_v5.34.10/lib/libTree.so
            #10 0x00007ffff4190c74 in TTree::CopyEntries(TTree*, long long, char const*) () from /opt/root/root_v5.34.10/lib/libTree.so
            #11 0x00007ffff418cd3f in TTree::CloneTree(long long, char const*) () from /opt/root/root_v5.34.10/lib/libTree.so
            #12 0x00007ffff4187b91 in TTree::Merge(TCollection*, TFileMergeInfo*) () from /opt/root/root_v5.34.10/lib/libTree.so
            #13 0x00007ffff41ad9d6 in ROOT::merge_TTree(void*, TCollection*, TFileMergeInfo*) () from /opt/root/root_v5.34.10/lib/libTree.so
            #14 0x00007ffff799c854 in TFileMerger::MergeRecursive(TDirectory*, TList*, int) () from /opt/root/root_v5.34.10/lib/libRIO.so
            #15 0x00007ffff799d98d in TFileMerger::PartialMerge(int) () from /opt/root/root_v5.34.10/lib/libRIO.so
            #16 0x0000000000401ea3 in main ()
            (gdb)

      What I tried so far:

      x) I used hadd -f2. That didn't help.

      x) I used hadd on another machine running with 5.30. There hadd worked without any error messages, but later when running over the files I got messages like this:
      "Error in <TBasket::Streamer>: The value of fKeylen is incorrect (-25585) ; trying to recover by setting it to zero
      Error in <TBasket::Streamer>: The value of fNevBufSize is incorrect (-1211332764) ; trying to recover by setting it to zero
      terminate called after throwing an instance of 'std::bad_alloc'
      what(): St9bad_alloc"

      Other information:

      The files where originally hbook files and got converted via h2root.

      Cheers, Robin Glattauer

        Attachments

        1. a1.root
          861 kB
        2. a2.root
          696 kB
        3. a3.root
          1.15 MB
        4. a4.root
          1.54 MB

          Activity

            People

            • Assignee:
              pcanal Philippe Canal
              Reporter:
              5c5771460c7e7f2ee8dd Robin Glattauer (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: