Uploaded image for project: 'CernVM'
  1. CernVM
  2. CVM-1618

mount process using 100% of cpu

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: High
    • Resolution: Cannot Reproduce
    • Affects Version/s: CernVM-FS 2.5
    • Fix Version/s: None
    • Component/s: CVMFS
    • Labels:
      None
    • Environment:
    • Platforms:
      x86_64-slc6-gcc48-opt
    • Development:

      Description

      When under heavy load (many SCORE jobs running) many Worker Nodes exhibit a condition where "top" shows one of the cvmfs mount processes using 100% of a cpu.  For example from last night:

      cvmfs 29761 1 3 Aug25 ? 02:36:10 /usr/bin/cvmfs2 -o rw,fsname=cvmfs2,allow_other,grab_mountpoint,uid=402,gid=401 atlas.cern.ch /cvmfs/atlas.cern.ch
      cvmfs 1032421 1 47 19:39 ? 00:47:29 /usr/bin/cvmfs2 -o rw,fsname=cvmfs2,allow_other,grab_mountpoint,uid=402,gid=401 cms.cern.ch /cvmfs/cms.cern.ch

      Other WN have shown different mount processes doing this.  The machine cpu load shows as VERY large, but no other processes are getting any run slices.

      The machine MAY also be reporting on the console
      audit: backlog limit exceeded

      In a typical situation, stopping HTCondor clears all of these symptoms, and the machine will go idle.  It is not unusual to see LOTS of repositories mounted (see list below).  Before putting the machine back in production, as a safety measure, I restart the autofs service.  The machine then will behave "normally", but future occurrences on the machine are not prohibited.

      I have a bug report tarball that I will attach.  The act of generating this tarball on the machine seems to have cleared all the cvmfs mounted repos.  This is from c-106-5.aglt2.org, which I will leave idle following the tarball generation so that any questions about it can be addressed.

       

      [root@bl-1-3 condor]# df -h|grep cvmfs
      /dev/sda2 26G 19G 5.4G 78% /var/cache/cvmfs2
      cvmfs2 21G 18G 2.6G 88% /cvmfs/config-osg.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/atlas.cern.ch
      cvmfs2 21G 18G 2.6G 88% /cvmfs/sft.cern.ch
      cvmfs2 21G 18G 2.6G 88% /cvmfs/cms.cern.ch
      cvmfs2 21G 18G 2.6G 88% /cvmfs/connect.opensciencegrid.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/gwosc.osgstorage.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/oasis.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/singularity.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/veritas.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/icecube.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/ligo-containers.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/nexo.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/snoplus.egi.eu
      cvmfs2 21G 18G 2.6G 88% /cvmfs/spt.opensciencegrid.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/stash.osgstorage.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/cms-ib.cern.ch
      cvmfs2 21G 18G 2.6G 88% /cvmfs/xenon.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/grid.cern.ch
      cvmfs2 21G 18G 2.6G 88% /cvmfs/fermilab.opensciencegrid.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/nova.osgstorage.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/des.osgstorage.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/mu2e.osgstorage.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/uboone.osgstorage.org
      cvmfs2 1000M 948M 53M 95% /cvmfs/minerva.osgstorage.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/annie.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/argoneut.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/cdf.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/cdms.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/coupp.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/d0.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/darkside.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/des.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/dune.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/gm2.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/icarus.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/lariat.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/minerva.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/minos.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/mu2e.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/nova.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/sbnd.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/seaquest.opensciencegrid.org
      cvmfs2 21G 18G 2.6G 88% /cvmfs/uboone.opensciencegrid.org

       

       

        Attachments

          Activity

            People

            • Assignee:
              jblomer Jakob Blomer
              Reporter:
              roball Robert Ball
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                PlannedEnd:
                PlannedStart: