Uploaded image for project: 'CernVM'
  1. CernVM
  2. CVM-1920

Fallback proxy order applied too late when squid not functioning

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Medium
    • Resolution: Completed
    • Affects Version/s: CernVM-FS 2.7.4
    • Fix Version/s: CernVM-FS 2.8
    • Component/s: CVMFS
    • Labels:
      None
    • Platforms:
      ANY
    • Development:

      Description

      I came across a case where CVMFS_HTTP_PROXY was configured to use a non-functioning squid, and even though it was a client in the middle U.S., it used the CERN backup proxies. I was able to reproduce it on my test machine. cvmfs_talk proxy info showed that the backup proxies were properly sorted such that FNAL backup came first, and it was the assigned proxy group, yet /var/log/messages showed it connecting to CERN:

      Load-balance groups:
      [0] http://131.225.152.161:8001 (squid.fnal.gov, +5h)
      [1] http://131.225.188.245:3126 (cmsextproxy.fnal.gov, +5h), http://131.225.188.246:3126 (cmsextproxy.fnal.gov, +5h)
      [2] http://128.142.168.202:3126 (cmsextproxy.cern.ch, +53s), http://128.142.33.31:3126 (cmsextproxy.cern.ch, +53s)
      Active proxy: [1] http://131.225.188.245:3126
      First fallback group: [1]
      

      Oct  7 14:41:38 fermicloud327 cvmfs2: (config-osg.opensciencegrid.org) switching proxy from http://131.225.152.161:8001 to http://128.142.33.31:3126
      Oct  7 14:41:38 fermicloud327 cvmfs2: (config-osg.opensciencegrid.org) geographic order of servers retrieved from cvmfs-s1bnl.opensciencegrid.org
      Oct  7 14:41:38 fermicloud327 cvmfs2: (config-osg.opensciencegrid.org) CernVM-FS: linking /cvmfs/config-osg.opensciencegrid.org to repository config-osg.opensciencegrid.org
      

      The problem is that it needs to switch to the fallback proxy to get the geo order, and at that point it couldn't yet sort them so it takes them in the order in which they are defined in the config file. Then after it has retrieved the order, it does not right away reset to use the sorted order.

      I guess that after 5 minutes when it retries the original squid it will at that point switch to the closer backup proxy.

      As a workaround, I plan to change the fallback proxy order in the OSG config repo, but that's not a complete solution. For one, it doesn't affect the config repo, and it takes a long time for everybody to upgrade that config rpm. Also, there are some users of the OSG config outside of North America.

        Attachments

          Activity

            People

            Assignee:
            dwd Dave Dykstra
            Reporter:
            dwd Dave Dykstra
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: