Stuck tracks in Polycones in ATLAS Simulation

The current ATLAS simulation has been trying to use VecGeom replacements for Cons, Polycons and Tubs in Geant4, but we've found a significantly higher fraction of stuck tracks compared to stock Geant4 in Polycone volumes. Studies show that this is reproducible across setups:

• Geant4 10.6.3 plus ATLAS patches (see https://gitlab.cern.ch/bmorgan/geant4/-/commits/v10.6.3.3-atlassim-5739/)
• Vecgeom 1.1.5 and VecGeom 1.1.20 (we're aware the latest version isn't nominally compatible with Geant4 10.6.3, but versions don't seem to change the problem we're seeing)
• Run2 and Run3 ATLAS geometries

With some advice from mnovak, we think we've confirmed this is not due to overlaps (the patched Geant4 includes outputs of the overlap check for stuck tracks). However, it's not obvious then what the problem is - it may be related to VECGEOM-598, though that was for a different shape. A log file (ATLAS-R3S-2021-02-00-00.log) from our reproducible test case is attached - here, a Geantino is started at the same position and direction as a gamma in the full Athena simulation that was found to trigger the issue, and gets stuck at the same position, resulting in a G4Exception. Full tracking verbosity was enabled, so it's quite detailed. The main stderr outputs are:

-------- EEEE ------- G4Exception-START -------- EEEE -------

*** G4Exception : GeomNav0003

      issued by : G4Navigator::ComputeStep()

Stuck Track: potential geometry or navigation problem.

  Track stuck, not moving for 25 steps.

  Current  phys volume: 'PixServMat1'

   - at position : (54.65773120144969,410.3569116640605,3450.000001134244)

     in direction: (0.1876537118052928,0.9065540604610374,0.3780817635212015)

    (local position: (382.7083757203281,157.8434720983534,-0.4524988657558424))

    (local direction: (0.8789257021658389,0.2907641486926912,0.3780817635212015)).

  Previous phys volume: 'InDetServMat'

  Likely geometry overlap - else navigation problem !

 Track *abandoned* due to excessive number of Zero steps. Event aborted. 

*** Event Must Be Aborted ***

G4WT0 > G4Track (0x7fd15e0d8f40) - track ID = 1, parent ID = 0

G4WT0 >  Particle type : geantino - creator process : not available

G4WT0 >  Kinetic energy : 1 GeV - Momentum direction : (0.187654,0.906554,0.378082)

G4WT0 >  Step length : 1 Ang - total energy deposit : 0 eV 

G4WT0 >  Pre-step point : (54.6577,410.357,3450) - Physical volume : PixServMat1 (pix::MatPP18)

G4WT0 >  - defined by : Transportation - step status : 1

G4WT0 >  Post-step point : (54.6577,410.357,3450) - Physical volume : PixServMat1 (pix::MatPP18)

G4WT0 >  - defined by : Transportation - step status : 7

G4WT0 >  *** Note: Step information might not be properly updated.

G4WT0 > 

-------- EEEE -------- G4Exception-END --------- EEEE -------

A bash script (ath-vecgeom-bug-builder.sh) is also attached that when run will checkout and build VecGeom, Geant4, and GeoModel(FullSimLight) with the same settings/options as used in ATLAS for CentOS7/GCC11. It will also generate a script that runs FullSimLight and that should reproduce the attached error shown in the log file. Instructions for use are documented at the top of the script. The geometry files can be downloaded from https://geomodel.web.cern.ch/atlas-geometry-data/ (CERN Authentication required), and the ATLAS-R3S-2021-02-00-00.db file can be used.

As I wear both ATLAS and VecGeom hats, I'm of course happy to help with further tests and working on any fixes needed on either side!