[SPI-969] multiprocessing python module from pytools incompatible with python 2.7 ? Created: 16/Mar/17  Updated: 30/Apr/21

Status: Reopened
Project: SPI
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: High
Reporter: Asoka De Silva Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

SLC6, Python 2.7


Customer: ATLAS
PlannedEnd:
Platforms:
x86_64-slc6-gcc48-opt
Development:
Actual Start:
Actual End:

 Description   

{{Hi,

The multiprocessor python found in pytools looks incompatible with that provided by python 27. (this is true also for LCG_88).

e.g.:

lcgenv -p LCG_88 x86_64-slc6-gcc49-opt pytools
> python
Python 2.7.13 (default, Mar 15 2017, 12:07:52)
[GCC 4.9.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> import imp
>>> imp.find_module('multiprocessing')
(None, '/cvmfs/sft.cern.ch/lcg/releases/LCG_88/multiprocessing/2.6.2.1/x86_64-slc6-gcc49-opt/lib/python2.7/site-packages/multiprocessing', ('', '', 5))
>>>

but if we do not set up pytools:

lcgenv -p LCG_88 x86_64-slc6-gcc49-opt Python
> python
Python 2.7.13 (default, Mar 15 2017, 12:07:52)
[GCC 4.9.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
>>>
>>> import imp
>>> imp.find_module('multiprocessing')
(None, '/cvmfs/sft.cern.ch/lcg/releases/LCG_88/Python/2.7.13/x86_64-slc6-gcc49-opt/lib/python2.7/multiprocessing', ('', '', 5))
>>>

regards,
Asoka}}



 Comments   
Comment by Patricia Mendez Lorenzo [ 17/Mar/17 ]

Hi Asoka,

this parameter is not supported with the current multiprocessing version we are running. This should be the reason. Actually If I do:

Help on function Pool in module multiprocessing:

Pool(processes=None, initializer=None, initargs=())
Returns a process pool object

nothing related to maxtasksperchild. But this is not something new, actually you had it also in LCG_87 which we run with the same version of multiprocessing and in all nightly builds available for testing before release

Cheers
Patricia

Comment by Asoka De Silva [ 17/Mar/17 ]

Hi Patricia,

However Python 2.7 supports it - e.g.

(on lxplus7 where python 2.7 is native)

[desilva@lxplus004 ~]$ python -V
Python 2.7.5
[desilva@lxplus004 ~]$ python
Python 2.7.5 (default, Nov 6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
>>> help(Pool)

shows

Help on function Pool in module multiprocessing:

Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None)
Returns a process pool object

so if the LCG multiprocessing module is present and in PYTHONPATH it will create confusion since it does not have the same arguments as the native python 2.7 version.

(I realize this is not something new but wanted to report it so that it can be addressed in newer LCG versions, if possible. We just have to handle through user support older / existing versions of LCG which are for example in ATLAS software.)

Thanks !

regards,
Asoka

Comment by Ivan Razumov [ 09/Apr/19 ]

Asoka De Silva As far as I can tell, this is no longer a problem: Python 2.7.15 supports maxtasksperchild argument. I'll close this Jira now, feel free to reopen it if necessary.

Comment by Omer Ogul Oncel [ 04/Sep/20 ]

Dear Asoka De Silva and Ivan Razumov,

I have recently come across a similar problem and ended up here upon searching for solutions. While working on LXPLUS, I encounter the "maxtasksperchild" error during the event generation with Madgraph at NLO, when using

set low_mem_multicore_nlo_generation True

command to speed up the process. Actually this problem is also discussed in a post on MG forum, albeit without a final conclusion, providing a link to this JIRA ticket here.

I can confirm that python 2.7.5 check Asoka did above is reproducible i.e.

 

[ooncel@lxplus797 ~]$ python -V
Python 2.7.5
[ooncel@lxplus797 ~]$ python
Python 2.7.5 (default, Apr 2 2020, 13:16:51) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
>>> help(Pool)
Help on function Pool in module multiprocessing:
Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None)
 Returns a process pool object
(END)

However, I cannot manage to find a version 2.7.15 mentioned by Ivan on last comment, perhaps its updated since a year has passed since then?

 

[ooncel@lxplus712 JO]$ showVersions python
python versions;
 --> 2.5.4p2-i686-slc5-gcc43 
 --> 2.5.4p2-x86_64-slc5-gcc43 
 --> 2.6.5-i686-slc5-gcc43 
 --> 2.6.5-x86_64-slc5-gcc43 
 --> 2.6.5p1-i686-slc5-gcc43 
 --> 2.6.5p1-x86_64-slc5-gcc43 
 --> 2.7.10-x86_64-slc6-gcc49 
 --> 2.7.10-x86_64-slc6-gcc62 
 --> 2.7.13-x86_64-slc6-gcc44 
 --> 2.7.13-x86_64-slc6-gcc49 
 --> 2.7.13-x86_64-slc6-gcc62 
 --> 2.7.3-i686-slc6-gcc47 
 --> 2.7.3-x86_64-slc6-gcc47 
 --> 2.7.4-x86_64-slc6-gcc48 default 
 --> 2.7.9-x86_64-slc6-gcc48 
 --> 2.7.9p1-x86_64-slc6-gcc49 
 --> 3.7.4-x86_64-centos7 pilot-default pilot-default-SL7 pilot-testing-SL7 pilotTest

 

And I have the feeling that problem might be related to the fact that setting up AthGeneration (for instance 21.6.40 that is reported to be working well with MG) also changes/overrides the python version used i.e. to 2.7.13 which seem to crash when one tries the check Asoka did with 2.7.5:

 

[ooncel@lxplus712 JO]$ source ../setuptest.sh 
lsetup lsetup (...)Using AthGeneration/21.6.40 [cmake] with platform x86_64-slc6-gcc62-opt
 at /cvmfs/atlas.cern.ch/repo/sw/software/21.6
[ooncel@lxplus712 JO]$ python -V
Python 2.7.13
[ooncel@lxplus712 JO]$ python 
Python 2.7.13 (default, Apr 22 2017, 20:06:00) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>>
 

I do not know if/how Madgraph can be forced to use a specified python version, or whether you have some alternative suggestions, perhaps a version which is know to be not affected?

Best regards,

Ogul

 

 

Comment by Ivan Razumov [ 07/Sep/20 ]

Dear Omer Ogul Oncel,

you can setup a different version of Python after setting up Madgraph by running the following command: source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh

Best regards,
Ivan

Comment by Omer Ogul Oncel [ 07/Sep/20 ]

Dear Ivan Razumov,

Thank you for your reply. I am able to setup the release, however, contrary to your previous check on 2.7.15, in my case the python check fails:

 

[ooncel@lxplus738 JO]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus738 JO]$ python -V
Python 2.7.15
[ooncel@lxplus738 JO]$ python
Python 2.7.15 (default, Feb 11 2019, 00:55:11) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> exit() 

I then tried Asoka's version of 2.7.5, but it is not available anymore, in agreement with the list of available setups shown on my previous post above.

 
[ooncel@lxplus738 JO]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.5/x86_64-slc6-gcc62-opt/Python-env.sh
-bash: /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.5/x86_64-slc6-gcc62-opt/Python-env.sh: No such file or directory

Therefore, I am still encountering the problem. Do you know of a version where this problem does not occur, or can you confirm that 2.7.15 is working for you and perhaps I am doing something wrong?

Best,

Ogul

Comment by Ivan Razumov [ 07/Sep/20 ]

Hi Omer Ogul Oncel,

first of all, it looks like you are mixing code built for slc6 and centos7 - you setup the environment for SLC6 (x86_64-slc6-gcc62-opt), but the lxplus machine you are using (lxplus738) runs CentOS 7. Second, even with this mix, I can't reproduce the issue you have:

razumov@lxplus738:/afs/cern.ch/user/r/razumov> source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh                                                                    
razumov@lxplus738:/afs/cern.ch/user/r/razumov> python -V
Python 2.7.15
razumov@lxplus738:/afs/cern.ch/user/r/razumov> python
Python 2.7.15 (default, Feb 11 2019, 00:55:11) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
>>> exit()

Could you please run the following commands and show the output after setting up python:

echo $PATH | sed -e "s/:/\n/g"
which python

Best,
Ivan

Comment by Omer Ogul Oncel [ 07/Sep/20 ]

Hi Ivan Razumov,

I think the problem with reproducibility is due to previous environment I setup. I am able to reproduce your situation when I just login to LXPLUS and do as you did:

[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Feb 11 2019, 00:55:11) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
>>> exit()

The actual environment I use has a preliminary setup.sh, as I previously mentioned, containing the AthGeneration setup i.e. 

setupATLAS
asetup 21.6.40,AthGeneration
# export ATHENA_PROC_NUMBER=10 Turned off for the moment

Which sets up slc6 automatically, therefore I was using that python setup as well.

Using AthGeneration/21.6.40 [cmake] with platform x86_64-slc6-gcc62-opt
 at /cvmfs/atlas.cern.ch/repo/sw/software/21.6

In conclusion the set of commands that should reproduce the crash is as follows:

 

[ooncel@lxplus740 ~]$ setupATLAS
[ooncel@lxplus740 ~]$ asetup 21.6.40,AthGeneration
Using AthGeneration/21.6.40 [cmake] with platform x86_64-slc6-gcc62-opt at /cvmfs/atlas.cern.ch/repo/sw/software/21.6
[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/Pythonenv.sh
[ooncel@lxplus740 ~]$ python -V
Python 2.7.15
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Feb 11 2019, 00:55:11) [GCC 6.2.0] on linux2Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last): 
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild' 

Following your request from there on, here is the output I get:

[ooncel@lxplus740 ~]$ echo $PATH | sed -e "s/:/\n/g"
/cvmfs/sft.cern.ch/lcg/releases/LCG_95/sqlite/3210000/x86_64-slc6-gcc62-opt/bin
/cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/bin
/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.28-a983d/x86_64-slc6/bin
/tmp/ooncel/.asetup-sysbin-5IeMWl_30098
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/share
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGenerationExternals/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/share
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGenerationExternals/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/xrootd/4.5.0/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/COOL/3_1_8/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/pytools/2.0/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/MCGenerators/sherpa/2.2.10p4/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/CORAL/3_1_8/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/Python/2.7.13/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/ROOT/6.08.06/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/graphviz/2.28.0/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/doxygen/1.8.11/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/sw/software/21.6/sw/lcg/releases/LCG_88/gperftools/2.5/x86_64-slc6-gcc62-opt/bin
/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/Cmake/3.11.0/Linux-x86_64/bin
/cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-b9934/x86_64-slc6/bin
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.28-a983d/x86_64-slc6/bin
/cvmfs/sft.cern.ch/lcg/releases/LCG_95/sqlite/3210000/x86_64-slc6-gcc62-opt/bin
/cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/bin
/afs/cern.ch/atlas/scripts
/usr/sue/bin
/usr/lib64/qt-3.3/bin
/usr/condabin
/usr/local/bin
/usr/bin
/usr/local/sbin
/usr/sbin
/opt/puppetlabs/bin
/afs/cern.ch/user/o/ooncel/bin
[ooncel@lxplus740 ~]$ which python
/cvmfs/sft.cern.ch/lcg/releases/LCG_95/Python/2.7.15/x86_64-slc6-gcc62-opt/bin/python

I hope these informations are helpful.

Best,

Ogul

Comment by Dmitri Konstantinov [ 07/Sep/20 ]

Hi, my "two cents". It seems that multiprocessing doesn't work in LGC_95.

Can you try to setup it from LCG_95?

Cheers,
Dmitri

Comment by Omer Ogul Oncel [ 07/Sep/20 ]

Hi Dmitri Konstantinov,

Thanks for "chipping in" and your suggestion. I have tried the following releases, listed as available for 2.7.15 [1], none of them seem to work after the AthGeneration setup.

Releases: 95, 95a, 95rc1, 94, 94a, 94rc1

[1]http://lcginfo.cern.ch/pkgver/Python/2.7.15/

Output:

 

[ooncel@lxplus740 ~]$ setupATLAS
[ooncel@lxplus740 ~]$ asetup 21.6.40,AthGenerationUsing AthGeneration/21.6.40 [cmake] with platform x86_64-slc6-gcc62-opt at /cvmfs/atlas.cern.ch/repo/sw/software/21.6
 
[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95rc1/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Nov 27 2018, 15:38:53) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> exit()
 
[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_95a/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Feb 11 2019, 00:55:11) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> exit()
 
[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_94a/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Jul 30 2018, 09:38:14) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> exit()
 
[ooncel@lxplus740 ~]$ source /cvmfs/sft.cern.ch/lcg/releases/LCG_94rc1/Python/2.7.15/x86_64-slc6-gcc62-opt/Python-env.sh
[ooncel@lxplus740 ~]$ python
Python 2.7.15 (default, Jul 30 2018, 09:38:14) 
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> x = Pool(maxtasksperchild=1)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Pool() got an unexpected keyword argument 'maxtasksperchild'
>>> 
>>> exit() 

 

Comment by Dmitri Konstantinov [ 07/Sep/20 ]

Hi Ivan, are you sure that "multiprocessing" works from any release?
When I setup env for python only - it works, but when I setup environment for "multiprocessing" it fails with this error mentioned in the ticket.

Comment by Dmitri Konstantinov [ 07/Sep/20 ]

Moreover, it seems that multiprocessing is a built-in. but we have it separately as well.
I tried Python from LCG_88 not having separate "multiprocessing" set up and it also works.

Comment by Ivan Razumov [ 07/Sep/20 ]

The multiprocessing package we have in our stack is a backport of Python 2.6's multiprocessing module to older Python versions. The maxtasksperchild option was introduced in Python 2.7 (https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool).

Comment by Omer Ogul Oncel [ 07/Sep/20 ]

Hi Dmitri Konstantinov,

For me LGC_88 also does not work after setting up AthGeneration i.e. asetup 21.6.40,AthGeneration

Ivan Razumov, I think this back port to 2.6 clarifies the root of the problem as MG authors also confirmed previously that the option would work only on Python 2.7. I was then confused as I am always setting up an 2.7 release and still getting a crash.

Is there a way to solve this issue then, or are there any plans/projects to update the multiprocessing module of python to 2.7?

Best,

Ogul

Comment by Ivan Razumov [ 07/Sep/20 ]

Hi Omer Ogul Oncel,

I have a new idea. Instead of setting up Python after setting up AthGenerators, could you please execute the following command: export PYTHONPATH="$(dirname $(which python))/../lib/python2.7:$PYTHONPATH" - this will make Python prioritize built-in multiprocessing over the pytools one.

Best,
Ivan

Comment by Omer Ogul Oncel [ 09/Sep/20 ]

Hi Ivan Razumov,

I am replying only now because I was trying to check your suggestion. The PYTHONPATH suggestion removed the original error I am having. However, I have a new "OSerror 39: directory not empty" problem now. I have tested a simple p p > t tbar [QCD] process where I turned-off low_memory_nlo command, with two setup cases to make sure the problem emerges from PYTHONPATH command.

Setup 1) 

 

setupATLAS
asetup 21.6.40,AthGeneration
export ATHENA_PROC_NUMBER=10

After this setup I run the code and get a normal output.

Setup 2)

 

setupATLAS
asetup 21.6.40,AthGeneration
export ATHENA_PROC_NUMBER=10
export PYTHONPATH="$(dirname $(which python))/../lib/python2.7:$PYTHONPATH" # Addition of this line

After this setup I run the code and get this error at the MadSpin decay part:

generate 09:33:48 Py:MadGraphUtils INFO Some errors detected by MadGraphControl - checking for serious errors
generate 09:33:48 Py:MadGraphUtils INFO stty: standard input: Inappropriate ioctl for device
generate 09:33:48 Py:MadGraphUtils INFO stty: standard input: Inappropriate ioctl for device
generate 09:33:48 Py:MadGraphUtils ERROR Command "launch --name=run_01" interrupted with error:
generate 09:33:48 Py:MadGraphUtils ERROR OSError : [Errno 39] Directory not empty: '/afs/cern.ch/work/o/ooncel/HNL_evtgen_JULY20/NLO_MASTER_JULY20/EVTGENAUG/JO/PROCNLO_SM_HeavyN_NLO_0/full_me/SubProcesses/P2_gg_ttxg_t_bwp_wp_udx_tx_bxwm_wm_dux'
generate 09:33:48 Py:MadGraphUtils ERROR Please report this bug on https://bugs.launchpad.net/mg5amcnlo
generate 09:33:48 Py:MadGraphUtils ERROR More information is found in '/afs/cern.ch/work/o/ooncel/HNL_evtgen_JULY20/NLO_MASTER_JULY20/EVTGENAUG/JO/PROCNLO_SM_HeavyN_NLO_0/run_01_tag_1_debug.log'.
generate 09:33:48 Py:MadGraphUtils ERROR Please attach this file to your report.
generate 09:33:48 Shortened traceback (most recent user call last):
generate 09:33:48 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/jobOptions/EvgenJobTransforms/skel.GENtoEVGEN.py", line 264, in <module>
generate 09:33:48 include(jofile)
generate 09:33:48 File "/afs/cern.ch/work/o/ooncel/HNL_evtgen_JULY20/NLO_MASTER_JULY20/EVTGENAUG/JO/000000/mc.MGPy8_2l_15_plus.py", line 6, in <module>
generate 09:33:48 include("MGPy8_2l_15_plus.py")
generate 09:33:48 File "/afs/cern.ch/work/o/ooncel/HNL_evtgen_JULY20/NLO_MASTER_JULY20/EVTGENAUG/JO/000000/MGPy8_2l_15_plus.py", line 199, in <module>
generate 09:33:48 generate(process_dir=process_dir,grid_pack=gridpack_mode,runArgs=runArgs,required_accuracy=required_accuracy)
generate 09:33:48 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/python/MadGraphControl/MadGraphUtils.py", line 393, in generate
generate 09:33:48 error_check(err)
generate 09:33:48 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.40/InstallArea/x86_64-slc6-gcc62-opt/python/MadGraphControl/MadGraphUtils.py", line 105, in error_check
generate 09:33:48 raise RuntimeError('Error detected in MadGraphControl process')
generate 09:33:48 RuntimeError: Error detected in MadGraphControl process
generate 09:33:48 Py:Athena INFO leaving with code 8: "an unknown exception occurred"

Looking around, I found this relevant bug report having same error on same process on same LXPLUS7 environment. It is not resolved though. Also, I fail to connect the two issues and come to conclusion. Perhaps it can help you figure out what is going on: https://bugs.launchpad.net/mg5amcnlo/+bug/1788615

Finally here is my setup if you like to test it: /afs/cern.ch/user/o/ooncel/work/public/SPI969

To run: (default setup.sh has PYTHONPATH command included)

 

source setup.sh
source generate.sh

 

Cheers,

Ogul

 

 

 

 

Comment by Ivan Razumov [ 24/Feb/21 ]

Since slc6 version of lxplus is no longer available, I tested the code inside docker container. After small fix to

{lhapdf-config}

(the original script found 2 copies of boost - slc6 and cenots7 ones, I hardcoded the path to point to slc6 one) the generation script worked fine. So I think the issue is somehow related to AFS.

Generated at Sat Sep 25 08:35:36 CEST 2021 using Jira 8.17.0#817000-sha1:a507a62ee263f31d253569e578e747c4fedadad0.