Details
-
Bug
-
Resolution: Fixed
-
Medium
-
CernVM-FS 2.3.2
-
None
-
RHEL 6.8
-
x86_64-slc6-gcc48-opt
-
Description
I found in testing server 2.3.2-1 that running a snapshot on certain repositories would fail "loading reflog" (8 - no text) and dump a core [1].
In these cases, the file '.cvmfsreflog' did exist and seemed like a valid SQLite file [2].
If I removed this reflog file and ran a snapshot, core dumped again. If I removed all .cvmfs* files, a snapshot ran and succeeded, but left behind another .cvmfsreflog file, and subsequent snapshots dumped cores.
The solution in all cases was to remove the reflog checksum file, which seems to be orphaned:
/var/spool/cvmfs/${repo}/reflog.chksum
|
The server package version is: cvmfs-server-2.3.2-1.el6.x86_64
I reported this to the Stratum One admin list, where Dave asked me to file a ticket. Unlike CVM-1114, I had not manually intervened on the system or run garbage collection. I have since cleaned up the core dumps and suspect files, but I assume this situation could be easily recreated with a stale or invalid reflog checksum.
[1]
# cvmfs_server snapshot ship.cern.ch
|
GeoIP Database is up to date (21 days old). Nothing to do.
|
CernVM-FS: replicating from http://cvmfs-stratum-zero.cern.ch:8000/cvmfs/ship.cern.ch
|
CernVM-FS: using public key(s) /etc/cvmfs/keys/cern.ch/cern.ch.pub, /etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub, /etc/cvmfs/keys/cern.ch/cern-it3.cern.ch.pub
|
failed loading reflog in 'ship.cern.ch' (8 - no text)
|
/usr/bin/cvmfs_server: line 5442: 4971 Aborted (core dumped) $user_shell "$(__swissknife_cmd dbg) pull -m $name -u $stratum0 -w $stratum1 -r ${upstream} -x ${spool_dir}/tmp -k $public_key -n $num_workers -t $timeout -a $retries $with_history $with_reflog $initial_snapshot_flag $timestamp_threshold $log_level
|
[2]
# file .cvmfsreflog
|
.cvmfsreflog: SQLite 3.x database
|
# strings .cvmfsreflog
|
SQLite format 3
|
Wtablerefsrefs
|
CREATE TABLE refs (hash TEXT, type INTEGER, timestamp INTEGER, CONSTRAINT pk_refs PRIMARY KEY (hash))'
|
indexsqlite_autoindex_refs_1refs
|
Atablepropertiesproperties
|
CREATE TABLE properties (key TEXT, value TEXT, CONSTRAINT pk_properties PRIMARY KEY (key))3
|
indexsqlite_autoindex_properties_1properties
|
%fqrnship.cern.ch
|
schema_revision0
|
schema1.0
|
fqrn
|
schema_revision
|
schema
|
814a6b91ac5ba51031ffcaf68c1db52be27b0eb8XYL
|
6fcbb0fa6b20bb707647d0ce495d54ea885aad25
|
814a6b91ac5ba51031ffcaf68c1db52be27b0eb8
|
] 6fcbb0fa6b20bb707647d0ce495d54ea885aad25
|
Attachments
Issue Links
- relates to
-
CVM-1124 Increase timeouts for garbage collection download manager
-
- Closed
-