[Long Running] Move artifacts to Object Storage

This issue describes steps to get to the point of migrating artifacts to object storage.

The migration is online, and not interrupting to existing users.

Configure

Make GitLab be aware of Object Storage as described in https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage,

How it works?

GitLab stores information where artifacts are stored: local or remote storage. By default all artifacts are stored locally, enabling object storage doesn't change that, artifacts are continued to be stored locally, but we gain an ability to migrate all existing artifacts to object storage and be served from there.

How to migrate artifacts:

Migrate all artifacts starting from oldest one, this is currently synchronous, single threaded process:

gitlab-rake gitlab:artifacts:migrate

Migrate specific artifact for the specific build (from rails/console):

Ci::Build.find(BUILD_ID).tap do |build|
  build.artifacts_file.migrate!(ArtifactUploader::REMOTE_STORE)
  build.artifacts_metadata.migrate!(ArtifactUploader::REMOTE_STORE)
end

Plan for making a change

1. Decide where we want to store artifacts, S3, Google (S3-compatible)?
2. Enable Object Storage in GitLab,
3. Migrate single build and tests everything for that project,
4. Migrate all artifacts for single project,
5. Migrate all artifacts for all projects,

What we may need to do?

1. We may need to improve the migration script to use threading approach to migrate multiple files at once,
2. For 10.0 implement an ability to automatically upload all new artifacts once uploaded,
3. For 10.1/2/3 make runner talk directly to object storage and reduce the egress traffic,

Who is gonna be involved from CI/CD team?

Initially @tmaczukin, later @ayufan.

What next?

Execute steps 1-3. If everything goes ok. Execute step 4 for gitlab-ce/www-gitlab-com, etc. If everything goes OK execute step 5.

cc @pcarranza @ernstvn

mentioned in issue #1666

changed the description

This is currently blocked by artifacts storage not working properly in Omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2640

@ayufan does this effort need attention from the Production team? If so, please cc @pcarranza and @sitschner

Yes. It does. We need to:

1. Create a new bucket to hold artifacts,
2. Configure production server,
3. Perform small-scale test (single artifact),
4. Perform mid-scale test (single project),
5. Run full-scale migration.

cc @sitschner @pcarranza

marked this issue as related to gitlab-org/omnibus-gitlab#2640 (closed)

@zj @ahanselka could you please work together on building a detailed plan of what can and can't be done and which are the next steps here?

@pcarranza I would be happy to. I would like to note that I am out next week and ZJ is out tomorrow, however. Thus we would not really be able to work together until the week of the 14th.

Created a toy project with artifacts we can try first: https://gitlab.com/zj/artifacts-moving-temp-project

If we have a bucket we should first move only those artifacts by doing:

project = Project.find_by_full_path('zj/artifacts-moving-temp-project')
#only one right now
project.builds.each do |build|
  build.artifacts_file.migrate!(ArtifactUploader::REMOTE_STORE)
  build.artifacts_metadata.migrate!(ArtifactUploader::REMOTE_STORE)
end

I doubt timing this will be useful, so this should only serve to check if this works. Than some manual check on the project.

@zj so just to be clear, right now GitLab only supports moving old artifacts to object storage for archival. New artifacts are still saved to the "local" disk and can be archived later, correct?

@ahanselka Correct, to show you:

the request comes in, on the runner API endpoint, which computes the location: in the uploader class. This uses local storage, indeed.

@zj Do we have to have gitlab_rails['artifacts_object_store_enabled'] = true in order to test the uploading? I ask because the docs say that enabling this will prevent browsing of artifacts via web, and that seems like something we wouldn't want to do. Relevant WIP MR is chef-repo!1014

@ahanselka This was the case some time, however, this is no longer the case for GitLab 9.5. I've opened an MR to reflect this.

First project was moved:

1. Moving went fine, but not tested at scale
2. Job view works fine
3. Browsing of artifacts worked :)
4. Downloading of the zip didn't work, both in the pipeline view and in the job view, the URL was mallformed.

This issue is now blocked until its resolved

Fix incoming by: gitlab-org/gitlab-ee!2705

@zj Excellent!

Also, what version are we targeting for being able to use solely object storage for artifacts? With the current implementation we will be able to archive all of our old artifacts and save lots of money by deleting the old artifact storage server, however ideally we will be able to bypass local storage altogether fairly soon.

@ahanselka I understand, however, I'm not aware of this being scheduled, so we'll have to wait until it is and will be implemented.

@mydigitalself how can we get this scheduled ASAP?

cc/ @sitschner

mentioned in issue #2524 (closed)

mentioned in issue #2526

@pcarranza @sitschner From CI team side I believe that it will happen with 10.2. In 10.1 we will make auto-archive to happen automatically. The 10.2 should bring built-in support in Runner to directly connect Object Storage and perform upload.

cc @bikebilly

@pcarranza we are dealing with the Platform aspects of Object Storage in 10.0 - anything to do with artifacts is @ayufan / @markpundsack / @bikebilly.

We're waiting on 9.5.1 to continue the the next round of testing.

The next target would be the gitlab-org group. This migration will run much longer, allowing us to determine how fast we're migrating the data to s3. Given this is a larger scale test we can than also determine the ingress/egress ratio. This might be ok now, or indicate we need to optimise our uploader.

Not sure if I can comment on this in the open, but our ingress can be viewed here

@bikebilly

It is about effort. Doing 1. is easy task, doing 2. is complex task.

We have begun moving old gitlab-org/gitlab-ce artifacts to S3. This is running on the deploy node in a tmux session. I have no idea how long it will take, but at an estimation of 100/minute, it will take roughly 48 hours to finish.

cc/ @zj

@ahanselka Can you update us on status?

At the time of writing we're transferring the www-gitlab-org, which actually is a tiny amount of artifacts. After which we'll push this a bit more aggresively: First we transfer everything created over 1 year ago. If those are done, every artifact create before May of this year.

So far our findings:

1. We try to move files which are expired by sidekiq already,
2. The sequential rate on the hours we ran this, is about 80 an hour.
3. So far no one complained about artifacts acting weird

Today we'll run:

require 'logger'

logger = Logger.new(STDOUT)
transferred = 0

# Mutes the AR queries being run
ActiveRecord::Base.logger = nil

Ci::Build.joins(:project).with_artifacts.where('created_at < ?', 1.year.ago).
  where(artifacts_file_store: [nil, ArtifactUploader::LOCAL_STORE]).
  find_each(batch_size: 100) do |build|
  begin
    build.artifacts_file.migrate!(ArtifactUploader::REMOTE_STORE)
    build.artifacts_metadata.migrate!(ArtifactUploader::REMOTE_STORE)

    transferred += 1
    logger.info "Migrated build ##{transferred} of about 300000" if transferred % 100 == 0
  rescue => e
    logger.error("Failed transfering #{build.id}: #{e.message}"
  end
end

Given the limitations and bounds of Ci::Builds to migrate, we can't yet use the rake task

I've begun the script and will check on it later. This will obviously take quite some time.

Progress update:

I, [2017-08-30T23:22:55.004969 #3158]  INFO -- : Migrated build #92900 of about 300000
I, [2017-08-30T23:23:25.987685 #3158]  INFO -- : Migrated build #93000 of about 300000
I, [2017-08-30T23:25:04.392878 #3158]  INFO -- : Migrated build #93100 of about 300000

@zj Things are moving along, but I checked just now and there are a few errors such as the following:

E, [2017-08-31T15:07:59.653785 #3158] ERROR -- : Failed transfering 2286926: getaddrinfo: Name or service not known (SocketError)
E, [2017-08-31T15:07:59.763166 #3158] ERROR -- : Failed transfering 2286930: getaddrinfo: Name or service not known (SocketError)
E, [2017-08-31T15:08:00.763608 #3158] ERROR -- : Failed transfering 2286949: getaddrinfo: Name or service not known (SocketError)
E, [2017-08-31T15:08:00.781340 #3158] ERROR -- : Failed transfering 2286953: getaddrinfo: Name or service not known (SocketError)
E, [2017-08-31T15:08:01.057677 #3158] ERROR -- : Failed transfering 2286966: getaddrinfo: Name or service not known (SocketError)

Is this something we need to do something about? The script itself is continuing and is currently at

I, [2017-08-31T15:24:29.652392 #3158]  INFO -- : Migrated build #172900 of about 300000

The socket error is quite interesting, I'll dig into that today. Especially given the Build IDs are quite close to each other.

assigned to @zj

Alrighty another update!

We had a few more errors, a sample of which is below:

E, [2017-09-01T15:52:13.446584 #3158] ERROR -- : Failed transfering 3055577: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-01T15:52:13.465610 #3158] ERROR -- : Failed transfering 3055579: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-01T15:52:13.498819 #3158] ERROR -- : Failed transfering 3055586: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-01T15:52:17.568352 #3158] ERROR -- : Failed transfering 3055597: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-01T15:52:17.621078 #3158] ERROR -- : Failed transfering 3055621: getaddrinfo: Name or service not known (SocketError)

And we are nearing the end of this run:

I, [2017-09-01T16:49:20.883265 #3158]  INFO -- : Migrated build #284600 of about 300000
I, [2017-09-01T16:50:53.376675 #3158]  INFO -- : Migrated build #284700 of about 300000
I, [2017-09-01T16:52:00.526847 #3158]  INFO -- : Migrated build #284800 of about 300000

cc/ @zj

@zj and I decided restarted the script using require 'resolv-replace', which should fix the resolution errors. Since Monday is a US holiday, this will give it plenty of time to transfer!

Given the resolver issue has been fixed, it seems, I'm unassigign myself.

Next batch will be started tuesday.

removed assignee

@zj I've just checked so that we can begin the next batch. We did still have a few name errors, even after adding the require 'resolv-replace'.

E, [2017-09-02T07:38:40.762392 #3158] ERROR -- : Failed transfering 3656704: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-02T07:38:40.781723 #3158] ERROR -- : Failed transfering 3656705: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-02T07:38:40.802153 #3158] ERROR -- : Failed transfering 3656706: getaddrinfo: Name or service not known (SocketError)
E, [2017-09-02T07:38:50.793500 #3158] ERROR -- : Failed transfering 3656711: getaddrinfo: Name or service not known (SocketError)

changed milestone to %WoW ending 2017-09-26

assigned to @ahanselka

changed milestone to %WoW ending 2017-09-19

We've decided to wait on continuing with the next batch until the resolver errors are... resolved .

@zj Has there been any progress made on resolving the resolver issues?

Could we run this process under strace and verify that gethostbyname isn't actually being called here?

@zj @ayufan and I did some troubleshooting on this today. We discovered that from time to time Amazon will return a SERVFAIL for the gitlab-artifacts.s3.amazonaws.com record. This causes the failure and is cached for about 60 seconds. We are discussion a solution, but this is definitely a problem on Amazon's side we will need to work around.

cc/ @northrup

Good find!

What is the TTL on that record? Shouldn't that DNS be cached for some time?

@stanhu Amazon sets their TTLs to 5 seconds. Thus, it will not be cached at all really.

@ahanselka Amazon, or Azure? I would expect the Azure to be problem.

@ayufan definitely Amazon. We do not use Azure's resolvers, we use our own. @northrup can probably give more details, but we don't think our server is the problem here.

Can we gather tcpdump from other end of our server? It seems kind of unexpected to blame Amazon for that failure :)

AWS networking :trollface:

mentioned in issue #2694 (closed)

mentioned in issue #2732 (closed)

I've restarted this after gitlab-com/infrastructure#2807. I will continue to monitor it as usual.

changed milestone to %WoW ending 2017-09-26

changed title from Move artifacts to Object Storage to [Long Running] Move artifacts to Object Storage

I have just checked up on the transfer. It is still going acceptably. There have been a few name resolution errors, however. I think it may be our server since we are doing so many queries.

@northrup is there rate limiting on our DNS servers?

mentioned in issue #2824 (closed)

I think it may be related to #2824 (closed).

Update: The file transfer is still going (yay!). We have so far moved 6.3TB total into S3! That's awesome!

The transfer is still going. We have transferred a total of 7.7 TB, thats 1.4 TB since my last updated. Not bad!

We are still getting the name errors, even after #2824 (closed) was resolved. As such, perhaps @northrup and I can meet about this on Monday. I tried to debug it myself, but I must be missing something.

@ahanselka I have send you a meeting invite for Monday to debug the DNS issue.

mentioned in issue #2850

marked this issue as related to #2850

@northrup Glorious! I look forward to it.

@zj for these failures should we be retrying? Independent of these lookup failures, the S3 api availability of S3 is not going to be high enough for there not to be occasional 500s anyway for puts.

@ahanselka @northrup Note that we saw a DNS error for a deploy today for packages-gitlab-com.s3-accelerate.amazonaws.com. https://gitlab.slack.com/archives/C101F3796/p1506329596000202 .

@jarv I'm trying to create a background worker that every hour queues N items left behind. This fixes a few things:

1. If the normal way to migrate, which we introduce for 10.1, fails, we don't have to retry

• We can't, AFAIK, control the back off period for Sidekiq between retries
• If the upstream provider is down, a minute later it probably still is, so the defaults are not good enough

2. This will migrate without the need of a rake task

• Just changing a setting in gitlab.rb will do

3. If we get the DNS resolving fails, we cache the result for 60 seconds, now we don't have to retry and can just wait.

• In case nearly all artifacts are migrated already, we only store artifacts for about an hour.

@jarv thanks for letting us know!

@zj So what is our expectation on migrating these artifacts? Am I clear in thinking your comment means that we won't be able to be confident that all artifacts have been migrated until 10.1 or later?

Update on the transfer:

@northrup and I were unable to meet about DNS due to the .com issues this morning and afternoon. We will be meeting tomorrow. As for the transfer, it is going on with a few name errors as expected. We have currently moved 11.1 TB into S3 (wow!).

@ahanselka

@zj So what is our expectation on migrating these artifacts? Am I clear in thinking your comment means that we won't be able to be confident that all artifacts have been migrated until 10.1 or later?

Re-run migration. It will pick what is left off. The process is idempotent.

changed milestone to %WoW ending 2017-10-03

The migration is ongoing. We have moved a total of 13.3 TB into S3. As an interesting side note, we have 1857359 objects in the artifacts bucket so far!

An update for you on this lovely Friday afternoon. The process is still moving as expected. Due to other fires and some time off, we still haven't uncovered the DNS issues fully. However, as @ayufan said, it is idempotent so we will be able to re-run this and catch all the failures once we fix DNS.

We have moved a total of 15.6TB of artifacts into S3 as of now, a total of 2016661 objects.

Wow! <3

Over the weekend, we've grown to 19TB and 2248761 objects. So awesome!

We currently have a total of 20.6TB and 2329665 objects.

added moved 1 moved 2 moved 3 moved 4 labels

changed milestone to %WoW

added moved 5 label

The transfer finished. Since obviously there were some failures, I'm going to restart the process. We still need to fix the DNS issues though so we can be certain everything is migrated.

We have 21.8 TB and 2411148 objects there so far.

@zj is there a way to see how many artifacts are left?

@zj To clarify, in the previous question I meant how many artifacts are left on disk within the group we've tried to move already.

@ahanselka We could write a query, but please lets talk about this a bit more before restarting. Next release will most probably include a migration that will automatically migrate all the artifacts, that way we don't have to monitor it anymore.

@zj alright, I moved some before I saw your comment, but I'll hold for further discussions.

23TB, 2496916 objects.

I look forward to moving forward on this.

@zj please schedule something on my calendar when you have time and are ready.

added blocked label