Uploaded image for project: 'ROOT'
  1. ROOT
  2. ROOT-10323

Segfault in PyROOT at exit

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.20/00
    • Component/s: PyROOT
    • Labels:
      None
    • Environment:

      LHCb nightly builds against LCG dev3 (ROOT master) builds

      Description

      We had segfaults at exit in Python scripts using PyROOT (cppyy) for quite some time, but only now I managed to look into them.

      This is a typical stack trace:

      #8  0x00007f81aa3f3060 in __memcmp_sse4_1 () from /lib64/libc.so.6
      #9  0x00007f81a9728cb4 in std::char_traits<char>::compare (__s1=0x4569497373656c34 <error: Cannot access memory at address 0x4569497373656c34>, __s2=0x7fffdf19f370 "HepMC", __n=5) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/char_traits.h:312
      #10 0x00007f81a972e521 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare (this=0x286eb10, __str=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/basic_string.h:2849
      #11 0x00007f81a972d999 in std::operator< <char, std::char_traits<char>, std::allocator<char> > (__lhs=..., __rhs=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/basic_string.h:6136
      #12 0x00007f81a972cc1b in std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator() (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=..., __y=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_function.h:386
      #13 0x00007f81a973a21d in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_lower_bound (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=0x286eaf0, __y=0x2c23750, __k=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_tree.h:1888
      #14 0x00007f81a9737cdd in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::find (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __k=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_tree.h:2539
      #15 0x00007f81a97366d9 in std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::find (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:776
      #16 0x00007f81a7465ca6 in TClassEdit::InsertStd[abi:cxx11](char const*) (tname=0x7fffdf19f510 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/foundation/src/TClassEdit.cxx:1871
      #17 0x00007f81a17657e3 in TClingClassInfo::TClingClassInfo (this=0xf359030, interp=0x2525190, name=0x7fffdf19f510 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TClingClassInfo.cxx:93
      #18 0x00007f81a177ccb4 in TCling::SetClassInfo (this=0x2523810, cl=0x339bbb0, reload=true) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:3703
      #19 0x00007f81a7488772 in TClass::SetUnloaded (this=0x339bbb0) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TClass.cxx:6071
      #20 0x00007f81a7424fa6 in ROOT::RemoveClass (cname=0x7f81992a26d4 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/cont/src/TClassTable.cxx:851
      #21 0x00007f81a9790be6 in ROOT::Internal::TDefaultInitBehavior::Unregister (this=0x7f81a7901310 <ROOT::Internal::DefineBehavior(void*, void*)::theDefault>, classname=0x7f81992a26d4 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/include/Rtypes.h:174
      #22 0x00007f81a74a43c8 in ROOT::TGenericClassInfo::~TGenericClassInfo (this=0x7f81994b2b60 <ROOT::GenerateInitInstanceLocal(HepMC::GenEvent const*)::instance>, __in_chrg=<optimized out>) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx:221
      #23 0x00007f81aa2c2c29 in __run_exit_handlers () from /lib64/libc.so.6
      #24 0x00007f81aa2c2c77 in exit () from /lib64/libc.so.6
      #25 0x00007f81aa2ab49c in __libc_start_main () from /lib64/libc.so.6
      #26 0x00000000004006be in _start ()
      

      After some debugging I managed to identify the cause as a weird order of operations at exti.
      In particular, the static storage used internally by the function TClassEdit::InsertStd (std::set<std::string> sSetSTLtypes) gets deleted before the call to ROOT::TGenericClassInfo::~TGenericClassInfo, which may invoke (indirectly) TClassEdit::InsertStd (via ROOT::Internal::TDefaultInitBehavior::Unregister).
      If we are lucky, the memory was not overwritten and the code goes through, but often it happens we are not lucky.

      I can reproduce the sequence with gdb (not the crash, because it depends on the memory layout):

      $ # on lxplus
      $ . /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-dbg/setup.sh
      $ gdb --args python -c 'import cppyy'
      GNU gdb (GDB) 8.3
      Copyright (C) 2019 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      Type "show copying" and "show warranty" for details.
      This GDB was configured as "x86_64-pc-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
          <http://www.gnu.org/software/gdb/documentation/>.
       
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from python...
      (gdb) set breakpoint pending on 
      (gdb) b 'std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()'
      Function "std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()" not defined.
      Breakpoint 1 ('std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()') pending.
      (gdb) b 'ROOT::TGenericClassInfo::~TGenericClassInfo()'
      Function "ROOT::TGenericClassInfo::~TGenericClassInfo()" not defined.
      Breakpoint 2 ('ROOT::TGenericClassInfo::~TGenericClassInfo()') pending.
      (gdb) r
      Starting program: /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-dbg/bin/python -c import\ cppyy
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      warning: File "/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-dbg/lib/libCore.so-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/".
      To enable execution of this file add
              add-auto-load-safe-path /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-dbg/lib/libCore.so-gdb.py
      line to your configuration file "~/.gdbinit".
      To completely disable this security protection add
              set auto-load safe-path /
      line to your configuration file "~/.gdbinit".
      For more information about this security protection see the
      "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
              info "(gdb)Auto-loading safe path"
      warning: File "/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/lib64/libstdc++.so.6.0.25-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/".
      [Detaching after fork from child process 16925]
      [Detaching after fork from child process 16927]
      [Detaching after fork from child process 16929]
      [Detaching after fork from child process 16931]
      [Detaching after fork from child process 16947]
      [Detaching after fork from child process 16949]
      [Detaching after fork from child process 16955]
       
      Breakpoint 1, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set (this=0x7fffee2e6440 <StdLen(std::basic_string_view<char, std::char_traits<char> >)::gInlined>, __in_chrg=<optimized out>)
          at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:281
      281           ~set() = default;
      (gdb) c
      Continuing.
       
      Breakpoint 1, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set (this=0x7fffee2e6520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __in_chrg=<optimized out>)
          at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:281
      281           ~set() = default;
      (gdb) 
      Continuing.
       
      Breakpoint 2, ROOT::TGenericClassInfo::~TGenericClassInfo (this=0x7ffff03a9960 <PyROOT::ROOT::GenerateInitInstance()::instance>, __in_chrg=<optimized out>)
          at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx:214
      214     /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx: No such file or directory.
      (gdb)
      

      What actually changes between a successful job and a possibly failing one is that in the latest some of the calls to ROOT::TGenericClassInfo::~TGenericClassInfo (those related to HepMC classes) happen between the destruction of sSetSTLtypes and the call to at_exit_of_TROOT() / TROOT::~TROOT().

        Attachments

          Activity

            People

            • Assignee:
              etejedor Enric Tejedor Saavedra
              Reporter:
              clemenci Marco Clemencic
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: