Skip to content
Snippets Groups Projects
New issue look: Off

Add LFS objects and attachments to the Geo testbed

    • Closed Issue created by Nick Thomas @nick.thomas

    Currently, LFS and attachment sync is not working on the testbed because there are no files on the Geo primary to sync.

    It would be good to add (a subset of?) these files to the primary so we can evaluate the sync mechanisms on the testbed. This will also help to reduce noise in the Geo and sidekiq logs.

    Screenshot_from_2017-09-19_16-54-14

    /cc @jarv

    Edited by Nick Thomas

    Designs

    Child items 0

    • No child items are currently open.

    Activity

    • Nick Thomas changed title from Add LFS objects and files to the Geo testbed to Add LFS objects and attachments to the Geo testbed

      changed title from Add LFS objects and files to the Geo testbed to Add LFS objects and attachments to the Geo testbed

    • Nick Thomas changed the description

      changed the description

    • John Jarvis assigned to @jarv

      assigned to @jarv

    • John Jarvis changed milestone to %WoW ending 2017-09-19

      changed milestone to %WoW ending 2017-09-19

    • John Jarvis added unscheduled label

      added unscheduled label

    • username-removed-274314 changed milestone to %WoW ending 2017-09-26

      changed milestone to %WoW ending 2017-09-26

    • John Jarvis changed milestone to %WoW

      changed milestone to %WoW

    • John Jarvis removed unscheduled label

      removed unscheduled label

    • John Jarvis changed milestone to %WoW ending 2017-10-03

      changed milestone to %WoW ending 2017-10-03

    • John Jarvis mentioned in issue #2897 (closed)

      mentioned in issue #2897 (closed)

    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      This is proving to be a bit difficult:

      • Needed to create disk snapshots of file-storage1
      • Restoring drives from snapshots and attaching doesn't work because there is a limit to how many disks you can attach
      • We can't simply start file-storage1 and mount it because it is on an overlapping subnet

      I will try to move the geo testbed to a new subnet, another option is to spin up a new VM and attach the disks but since file-storage1 is running I will try this first.

    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      I've re-animated the old file-storage1 server and mounted it under /var/opt/gitlab/gitlab-rails/ which gives us:

      /var/opt/gitlab/gitlab-rails/shared/lfs-objects	
/var/opt/gitlab/gitlab-rails/uploads

      on the primary.

      @nick.thomas - in theory should geo start trying to sync these automatically or will we need to do something to restart it?

    • Nick Thomas
      Nick Thomas @nick.thomas ·
      Author Developer

      Thanks @jarv . I can see attempts to sync files in the geo.log:

      

{"severity":"INFO","time":"2017-10-03T14:30:43.551Z","class":"Geo::FileDownloadDispatchWorker","message":"Loop 958","enqueued":10,"pending":507,"scheduled":10}

{"severity":"INFO","time":"2017-10-03T14:30:43.646Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674819,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.118}

{"severity":"INFO","time":"2017-10-03T14:30:43.666Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559598,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.137}

{"severity":"INFO","time":"2017-10-03T14:30:43.673Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674817,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.143}

{"severity":"INFO","time":"2017-10-03T14:30:43.687Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559597,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.156}

{"severity":"INFO","time":"2017-10-03T14:30:43.694Z","class":"Geo::FileDownloadService","object_type":"avatar","object_db_id":674816,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.159}

{"severity":"INFO","time":"2017-10-03T14:30:43.709Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559595,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.164}

{"severity":"INFO","time":"2017-10-03T14:30:43.718Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559596,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.171}

{"severity":"INFO","time":"2017-10-03T14:30:43.722Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674818,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.186}

{"severity":"INFO","time":"2017-10-03T14:30:43.724Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674815,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.153}

{"severity":"INFO","time":"2017-10-03T14:30:43.731Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559594,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.159}

# ...



{"severity":"INFO","time":"2017-10-03T14:33:21.661Z","class":"Geo::FileDownloadDispatchWorker","message":"Loop 1109","enqueued":10,"pending":229,"scheduled":10}

{"severity":"INFO","time":"2017-10-03T14:33:21.698Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559459,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.112}

{"severity":"INFO","time":"2017-10-03T14:33:21.710Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559458,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.119}

{"severity":"INFO","time":"2017-10-03T14:33:21.785Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674663,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.193}

{"severity":"INFO","time":"2017-10-03T14:33:21.986Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559457,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.357}

{"severity":"INFO","time":"2017-10-03T14:33:22.087Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674664,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.401}

{"severity":"INFO","time":"2017-10-03T14:33:22.087Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674661,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.359}

{"severity":"INFO","time":"2017-10-03T14:33:22.185Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559456,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.532}

{"severity":"INFO","time":"2017-10-03T14:33:22.185Z","class":"Geo::FileDownloadService","object_type":"file","object_db_id":674662,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.569}

{"severity":"INFO","time":"2017-10-03T14:33:22.191Z","class":"Geo::FileDownloadService","object_type":"lfs","object_db_id":2559455,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.526}

{"severity":"INFO","time":"2017-10-03T14:33:22.390Z","class":"Geo::RepositorySyncService","project_id":1289258,"project_path":"librehealth/community-design-assets","message":"Trying to obtain lease to sync repository"}

{"severity":"INFO","time":"2017-10-03T14:33:22.400Z","class":"Geo::FileDownloadService","object_type":"avatar","object_db_id":674660,"message":"File download","success":false,"bytes_downloaded":-1,"download_time_s":0.727}

      So far nothing has replicated successfully. I'll leave it another couple of hours then investigate in more detail.

      For now, it seems prod is still giving 404 responses to sync:

      Gitlab::Geo::LfsTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::LfsTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::FileTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::FileTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::FileTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::LfsTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::LfsTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::FileTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::LfsTransfer: Unsuccessful download: 404 Not Found
Gitlab::Geo::FileTransfer: Unsuccessful download: 404 Not Found

      (in sync.geo's production.log due to an open bug)

      Edited by Nick Thomas
    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      Hm, this is strange considering we should have the full set of data from production.

    • Nick Thomas
      Nick Thomas @nick.thomas ·
      Author Developer

      @jarv some are definitely missing - consider the file referenced by LfsObject.find(2559490), for instance. It's not on the filesystem on prod.

    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      After discussing this over with @ahanselka it appears that this file-storage1 is very old for lfs objects which is why we are seeing many missing. I am going to spin up new servers and with snapshots of the current prod data to ensure we are up-to-date so unfortunately this isn't done quite yet.

    • username-removed-274314 changed milestone to %WoW

      changed milestone to %WoW

    • username-removed-274314 added moved 1 label

      added moved 1 label

    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      For this i will look for snapshots of:

      • uploads (nfs-uploads-01)
      • lfs (nfs-lfs-01)
    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      @nick.thomas OK we now have up-to-date copies of lfs and uploads

      irb(main):002:0> LfsObject.find(2559490).file_url
=> "/var/opt/gitlab/gitlab-rails/shared/lfs-objects/fb/3f/471d03cc84d40268d75a02844530e9b2e1602a66f341eca5b92bb77d3c53"

root@prod:~# ls -la  /var/opt/gitlab/gitlab-rails/shared/lfs-objects/fb/3f/471d03cc84d40268d75a02844530e9b2e1602a66f341eca5b92bb77d3c53
-rw-r--r-- 1 git git 1324846 Aug 10 20:16 /var/opt/gitlab/gitlab-rails/shared/lfs-objects/fb/3f/471d03cc84d40268d75a02844530e9b2e1602a66f341eca5b92bb77d3c53
    • John Jarvis
      John Jarvis @jarv ·
      Maintainer

      Closing, please feel free to re-open if you think there is more to be done.

    • John Jarvis closed

      closed

    Please register or sign in to reply