Details
-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 6.20/00
-
Component/s: PyROOT
-
Labels:None
-
Environment:
LHCb nightly builds against LCG dev3 (ROOT master) builds
Description
We had segfaults at exit in Python scripts using PyROOT (cppyy) for quite some time, but only now I managed to look into them.
This is a typical stack trace:
#8 0x00007f81aa3f3060 in __memcmp_sse4_1 () from /lib64/libc.so.6
|
#9 0x00007f81a9728cb4 in std::char_traits<char>::compare (__s1=0x4569497373656c34 <error: Cannot access memory at address 0x4569497373656c34>, __s2=0x7fffdf19f370 "HepMC", __n=5) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/char_traits.h:312
|
#10 0x00007f81a972e521 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare (this=0x286eb10, __str=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/basic_string.h:2849
|
#11 0x00007f81a972d999 in std::operator< <char, std::char_traits<char>, std::allocator<char> > (__lhs=..., __rhs=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/basic_string.h:6136
|
#12 0x00007f81a972cc1b in std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator() (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=..., __y=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_function.h:386
|
#13 0x00007f81a973a21d in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_lower_bound (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=0x286eaf0, __y=0x2c23750, __k=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_tree.h:1888
|
#14 0x00007f81a9737cdd in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::find (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __k=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_tree.h:2539
|
#15 0x00007f81a97366d9 in std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::find (this=0x7f81a7917520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __x=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:776
|
#16 0x00007f81a7465ca6 in TClassEdit::InsertStd[abi:cxx11](char const*) (tname=0x7fffdf19f510 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/foundation/src/TClassEdit.cxx:1871
|
#17 0x00007f81a17657e3 in TClingClassInfo::TClingClassInfo (this=0xf359030, interp=0x2525190, name=0x7fffdf19f510 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TClingClassInfo.cxx:93
|
#18 0x00007f81a177ccb4 in TCling::SetClassInfo (this=0x2523810, cl=0x339bbb0, reload=true) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/metacling/src/TCling.cxx:3703
|
#19 0x00007f81a7488772 in TClass::SetUnloaded (this=0x339bbb0) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TClass.cxx:6071
|
#20 0x00007f81a7424fa6 in ROOT::RemoveClass (cname=0x7f81992a26d4 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/cont/src/TClassTable.cxx:851
|
#21 0x00007f81a9790be6 in ROOT::Internal::TDefaultInitBehavior::Unregister (this=0x7f81a7901310 <ROOT::Internal::DefineBehavior(void*, void*)::theDefault>, classname=0x7f81992a26d4 "HepMC::GenEvent") at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT-HEAD-build/include/Rtypes.h:174
|
#22 0x00007f81a74a43c8 in ROOT::TGenericClassInfo::~TGenericClassInfo (this=0x7f81994b2b60 <ROOT::GenerateInitInstanceLocal(HepMC::GenEvent const*)::instance>, __in_chrg=<optimized out>) at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx:221
|
#23 0x00007f81aa2c2c29 in __run_exit_handlers () from /lib64/libc.so.6
|
#24 0x00007f81aa2c2c77 in exit () from /lib64/libc.so.6
|
#25 0x00007f81aa2ab49c in __libc_start_main () from /lib64/libc.so.6
|
#26 0x00000000004006be in _start ()
|
After some debugging I managed to identify the cause as a weird order of operations at exti.
In particular, the static storage used internally by the function TClassEdit::InsertStd (std::set<std::string> sSetSTLtypes) gets deleted before the call to ROOT::TGenericClassInfo::~TGenericClassInfo, which may invoke (indirectly) TClassEdit::InsertStd (via ROOT::Internal::TDefaultInitBehavior::Unregister).
If we are lucky, the memory was not overwritten and the code goes through, but often it happens we are not lucky.
I can reproduce the sequence with gdb (not the crash, because it depends on the memory layout):
$ # on lxplus
|
$ . /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-dbg/setup.sh
|
$ gdb --args python -c 'import cppyy'
|
GNU gdb (GDB) 8.3
|
Copyright (C) 2019 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law.
|
Type "show copying" and "show warranty" for details.
|
This GDB was configured as "x86_64-pc-linux-gnu".
|
Type "show configuration" for configuration details.
|
For bug reporting instructions, please see:
|
<http://www.gnu.org/software/gdb/bugs/>.
|
Find the GDB manual and other documentation resources online at:
|
<http://www.gnu.org/software/gdb/documentation/>.
|
|
|
For help, type "help".
|
Type "apropos word" to search for commands related to "word"...
|
Reading symbols from python...
|
(gdb) set breakpoint pending on
|
(gdb) b 'std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()'
|
Function "std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()" not defined.
|
Breakpoint 1 ('std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set()') pending.
|
(gdb) b 'ROOT::TGenericClassInfo::~TGenericClassInfo()'
|
Function "ROOT::TGenericClassInfo::~TGenericClassInfo()" not defined.
|
Breakpoint 2 ('ROOT::TGenericClassInfo::~TGenericClassInfo()') pending.
|
(gdb) r
|
Starting program: /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc8-dbg/bin/python -c import\ cppyy
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib64/libthread_db.so.1".
|
warning: File "/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-dbg/lib/libCore.so-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/".
|
To enable execution of this file add
|
add-auto-load-safe-path /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-dbg/lib/libCore.so-gdb.py
|
line to your configuration file "~/.gdbinit".
|
To completely disable this security protection add
|
set auto-load safe-path /
|
line to your configuration file "~/.gdbinit".
|
For more information about this security protection see the
|
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
|
info "(gdb)Auto-loading safe path"
|
warning: File "/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/lib64/libstdc++.so.6.0.25-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/".
|
[Detaching after fork from child process 16925]
|
[Detaching after fork from child process 16927]
|
[Detaching after fork from child process 16929]
|
[Detaching after fork from child process 16931]
|
[Detaching after fork from child process 16947]
|
[Detaching after fork from child process 16949]
|
[Detaching after fork from child process 16955]
|
|
|
Breakpoint 1, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set (this=0x7fffee2e6440 <StdLen(std::basic_string_view<char, std::char_traits<char> >)::gInlined>, __in_chrg=<optimized out>)
|
at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:281
|
281 ~set() = default;
|
(gdb) c
|
Continuing.
|
|
|
Breakpoint 1, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~set (this=0x7fffee2e6520 <TClassEdit::InsertStd[abi:cxx11](char const*)::sSetSTLtypes>, __in_chrg=<optimized out>)
|
at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/stl_set.h:281
|
281 ~set() = default;
|
(gdb)
|
Continuing.
|
|
|
Breakpoint 2, ROOT::TGenericClassInfo::~TGenericClassInfo (this=0x7ffff03a9960 <PyROOT::ROOT::GenerateInitInstance()::instance>, __in_chrg=<optimized out>)
|
at /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx:214
|
214 /mnt/build/jenkins/workspace/build/projects/ROOT-HEAD/src/ROOT/HEAD/core/meta/src/TGenericClassInfo.cxx: No such file or directory.
|
(gdb)
|
What actually changes between a successful job and a possibly failing one is that in the latest some of the calls to ROOT::TGenericClassInfo::~TGenericClassInfo (those related to HepMC classes) happen between the destruction of sSetSTLtypes and the call to at_exit_of_TROOT() / TROOT::~TROOT().