CVM-1905, a cvmfs_server check -r is not repairing anything. It just reports "no problems found". I also don't understand how the repositories got into this situation. There is no reflog.chksum nor .cvmfsreflog. These machines are each hosting 2 repositories for publication and two for replication, and the ones having problems are the ones for publication; the ones doing replication do have reflog.chksum and .cvmfsreflog files and are fine. The machines were both rebooted yesterday morning and started failing the first publish after the reboot. In fact I now see the development machine has the same issue.
I did an strace on one of the publish commands and see it creating reflog.chksum but it is not there when the publish command stops. I will attach the strace starting from the cvmfs_swissknife command (otherwise the file was too big). I will also attach a bugreport tarball, even though I know that's mostly for diagnosing client issues.
Doing the other workaround I mentioned in
CVM-1905, to put an "x" in the reflog.chksum file, did cause it to start working, but this is not a good situation. How do we prevent it from happening again?