Uploaded image for project: 'CernVM'
  1. CernVM
  2. CVM-1145

reflog replication errors and core dumps

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Medium
    • Resolution: Fixed
    • CernVM-FS 2.3.2
    • CernVM-FS 2.3.3
    • CVMFS
    • None
    • RHEL 6.8

    • x86_64-slc6-gcc48-opt

    Description

      I found in testing server 2.3.2-1 that running a snapshot on certain repositories would fail "loading reflog" (8 - no text) and dump a core [1].

      In these cases, the file '.cvmfsreflog' did exist and seemed like a valid SQLite file [2].

      If I removed this reflog file and ran a snapshot, core dumped again. If I removed all .cvmfs* files, a snapshot ran and succeeded, but left behind another .cvmfsreflog file, and subsequent snapshots dumped cores.

      The solution in all cases was to remove the reflog checksum file, which seems to be orphaned:

      /var/spool/cvmfs/${repo}/reflog.chksum

      The server package version is: cvmfs-server-2.3.2-1.el6.x86_64

      I reported this to the Stratum One admin list, where Dave asked me to file a ticket. Unlike CVM-1114, I had not manually intervened on the system or run garbage collection. I have since cleaned up the core dumps and suspect files, but I assume this situation could be easily recreated with a stale or invalid reflog checksum.

      [1]

      # cvmfs_server snapshot ship.cern.ch
      GeoIP Database is up to date (21 days old). Nothing to do.
      CernVM-FS: replicating from http://cvmfs-stratum-zero.cern.ch:8000/cvmfs/ship.cern.ch
      CernVM-FS: using public key(s) /etc/cvmfs/keys/cern.ch/cern.ch.pub, /etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub, /etc/cvmfs/keys/cern.ch/cern-it3.cern.ch.pub
      failed loading reflog in 'ship.cern.ch' (8 - no text)
      /usr/bin/cvmfs_server: line 5442:  4971 Aborted                 (core dumped) $user_shell "$(__swissknife_cmd dbg) pull -m $name         -u $stratum0                                           -w $stratum1 -r ${upstream}                  -x ${spool_dir}/tmp -k $public_key                                         -n $num_workers                                      -t $timeout                     -a $retries $with_history $with_reflog       $initial_snapshot_flag $timestamp_threshold $log_level

      [2]

      # file .cvmfsreflog
      .cvmfsreflog: SQLite 3.x database
      # strings .cvmfsreflog
      SQLite format 3
      Wtablerefsrefs
      CREATE TABLE refs (hash TEXT, type INTEGER, timestamp INTEGER, CONSTRAINT pk_refs PRIMARY KEY (hash))'
      indexsqlite_autoindex_refs_1refs
      Atablepropertiesproperties
      CREATE TABLE properties (key TEXT, value TEXT, CONSTRAINT pk_properties PRIMARY KEY (key))3
      indexsqlite_autoindex_properties_1properties
      %fqrnship.cern.ch
      schema_revision0
      schema1.0
      fqrn
      schema_revision
      	schema
      814a6b91ac5ba51031ffcaf68c1db52be27b0eb8XYL
      6fcbb0fa6b20bb707647d0ce495d54ea885aad25
      814a6b91ac5ba51031ffcaf68c1db52be27b0eb8
      ]	6fcbb0fa6b20bb707647d0ce495d54ea885aad25

      Attachments

        Issue Links

          Activity

            People

              jblomer Jakob Blomer
              destefan John Steven De Stefano Jr
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Actual End: