Uploaded image for project: 'CernVM'
  1. CernVM
  2. CVM-881

cvmfs_server snapshot storing to Amazon S3 does not clean up tmp files

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Medium
    • Resolution: Clarified
    • Affects Version/s: CernVM-FS 2.1.20
    • Fix Version/s: CernVM-FS 2.2.0
    • Component/s: CVMFS
    • Labels:
      None
    • Platforms:
      x86_64-slc6-gcc48-opt

      Description

      cvmfs-server-2.1.20's cvmfs_server snapshot when uploading to Amazon S3 does not clean up cvmfs.* files in /var/spool/cvmfs/<repo>/tmp. Reproduce with:
      1. create /etc/cvmfs/s3.conf with contents like this:

          CVMFS_S3_ACCOUNTS=1
          CVMFS_S3_ACCESS_KEY=youraccesskey
          CVMFS_S3_SECRET_KEY=yoursecretkey
          CVMFS_S3_BUCKETS_PER_ACCOUNT=1
          CVMFS_S3_BUCKET=cvmfs-stratum1
          CVMFS_S3_HOST=s3-us-west-2.amazonaws.com
          CVMFS_S3_PORT=80
          CVMFS_S3_MAX_NUMBER_OF_PARALLEL_CONNECTIONS=32

      2. create replica like this:

          # source /etc/cvmfs/s3.conf
          # cvmfs_server add-replica -u S3,/tmp,cvmfs/config-egi.egi.eu@/etc/cvmfs/s3.conf -w http://$CVMFS_S3_BUCKET-1-1.$CVMFS_S3_HOST /cvmfs/config-egi.egi.eu -o root http://cvmfs-stratum0.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu /etc/cvmfs/keys/egi.eu/egi.eu.pub

      3. then every time you run cvmfs_server snapshot, cvmfs.* files are in /var/spool/cvmfs/config-egi.egi.eu/tmp.

        Attachments

          Activity

          Hide
          rmeusel Rene Meusel (Inactive) added a comment -

          On first glance, I cannot reproduce this behaviour on the latest devel branch. The cvmfs.XXXXX files are created and properly cleaned afterwards. Could be that this was implicitly fixed with CVM-818. I'm having a deeper look to confirm that.

          Show
          rmeusel Rene Meusel (Inactive) added a comment - On first glance, I cannot reproduce this behaviour on the latest devel branch. The cvmfs.XXXXX files are created and properly cleaned afterwards. Could be that this was implicitly fixed with CVM-818 . I'm having a deeper look to confirm that.
          Hide
          rmeusel Rene Meusel (Inactive) added a comment -

          Regression testing on 2.1.20 is reproducing the issue. Must have been fixed implicitly... Checking.

          Show
          rmeusel Rene Meusel (Inactive) added a comment - Regression testing on 2.1.20 is reproducing the issue. Must have been fixed implicitly... Checking.
          Hide
          rmeusel Rene Meusel (Inactive) added a comment -

          The S3 implementation doesn't properly report the path of the uploaded local file (i.e. cvmfs.XXXXX) back to the caller. Therefore, the responsible callback cannot clean up the temporary files when using the S3 backend implementation. Unfortunately the failing unlink() call is ignored which made the problem hard to track down.

          The latest devel-branch contains some changes to the S3 implementation that properly report the path of these temporary files both in the S3 and local backends. Hence, the next release of CVMFS will fix the problem.

          Show
          rmeusel Rene Meusel (Inactive) added a comment - The S3 implementation doesn't properly report the path of the uploaded local file (i.e. cvmfs.XXXXX ) back to the caller. Therefore, the responsible callback cannot clean up the temporary files when using the S3 backend implementation. Unfortunately the failing unlink() call is ignored which made the problem hard to track down. The latest devel-branch contains some changes to the S3 implementation that properly report the path of these temporary files both in the S3 and local backends. Hence, the next release of CVMFS will fix the problem.
          Hide
          dwd Dave Dykstra added a comment -

          Great, thanks for investigating, Rene. Next time I need to upload to S3 I'll try using a development build.

          Show
          dwd Dave Dykstra added a comment - Great, thanks for investigating, Rene. Next time I need to upload to S3 I'll try using a development build.

            People

            • Assignee:
              rmeusel Rene Meusel (Inactive)
              Reporter:
              dwd Dave Dykstra
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: