Artifact size limit on GitLab.com is causing gitlab-org/gitlab-ce jobs to fail

e.g. https://gitlab.com/gitlab-org/gitlab-ce/pipelines/12487550

I don't know what the current size limit is, or what the artifact size will be, but can we up the limit to get pipelines working again and then consider how to reduce the artifact size for gitlab-ce if necessary?

@nick.thomas could this be a setting reset? where is the artifact limit defined?

@pcarranza it's in Application Settings. I've noticed that this has happened just after the 10.0.3 deploy, so it might be a newly-introduced bug rather than a genuine limit overreach.

yeah, this is impacting again the storage configuration :(

The problem is duplicate ApplicationSetting:

irb(main):004:0> ApplicationSetting.pluck(:created_at)
=> [Thu, 05 Oct 2017 14:41:32 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:35 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:34 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:38 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Fri, 16 Jan 2015 10:40:35 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00, Thu, 05 Oct 2017 14:41:33 UTC +00:00]

I found repository storage paths had been reset to default (single nfs server)

I reset to multiple servers

Related issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/34728

added app bug oncall labels

@sitschner I think that's a consequence of the application_settings duplicates - it's the list of enabled shards stored in application_settings.repository_storages, which defaults to ['default'] (the full list of shards comes from the config file).

It should be safe to delete all the new rows that were created at deploy time, just leaving the original settings untouched.

We should check dev again to make sure that's still got the expected settings.

We reverted to defaults for all settings. Email confirmation was disabled, recaptcha, storage sizes, sentry, prometheus, influx, etc all disabled.

I've re-enabled email confirmation but we really need to revert to an old settings copy.

Here are the unique created_at dates from @ayufan's snippet above:

=> ["Thu, 05 Oct 2017 14:41:32 UTC",
 "Thu, 05 Oct 2017 14:41:33 UTC",
 "Thu, 05 Oct 2017 14:41:35 UTC",
 "Thu, 05 Oct 2017 14:41:34 UTC",
 "Thu, 05 Oct 2017 14:41:38 UTC",
 "Fri, 16 Jan 2015 10:40:35 UTC"]

I strongly believe that Fri, 16 Jan 2015 10:40:35 UTC is the row we need, and all others can be summarily removed.

@nick.thomas

# SELECT created_at FROM application_settings WHERE id='1';
         created_at         
----------------------------
 2015-01-16 10:40:35.266484

All the rest were created later.

I've restored the domain blacklists by reading the old settings and importing them as a file.

What was our artifact limit set to before? I see it as 1000 MB.

Our blog post is blocked due to hitting this artifact limit: https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/35242333

I don't get it. The api_json.log from https://log.gitlap.com/goto/bf10c73499137b0b0f001a4a515e447f shows:

{  
   "time":"2017-10-05T17:19:34.995Z",
   "severity":"INFO",
   "duration":22.04,
   "db":3.33,
   "view":18.71,
   "status":200,
   "method":"POST",
   "path":"/api/v4/jobs/35242333/artifacts/authorize",
   "params":{  
      "expire_in":"7 days"
   },
   "host":"gitlab.com",
   "ip":"x x",
   "ua":"gitlab-runner 10.0.0-rc.1 (master; go1.8.3; linux/amd64)"
}

AND:

  
   "time":"2017-10-05T17:19:39.675Z",
   "severity":"INFO",
   "duration":19.43,
   "db":4.32,
   "view":15.11,
   "status":413,
   "method":"POST",
   "path":"/api/v4/jobs/35242333/artifacts",
   "params":{  
      "expire_in":"7 days",
      "file.path":"/var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/multipart-162949970/artifacts.zip",
      "file.name":"artifacts.zip",
      "metadata.path":"/var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/metadata_581076617",
      "metadata.name":"metadata.gz"
   },
   "host":"gitlab.com",
   "ip":"x x",
   "ua":"gitlab-runner 10.0.0-rc.1 (master; go1.8.3; linux/amd64)"
}

I don't see the filesize parameter here at all.

Oh, we must be hitting this line: https://gitlab.com/gitlab-org/gitlab-ee/blob/v10.0.3-ee/lib/api/runner.rb#L223

I've increased the limit to 2000 MB and retried the job above. Let's see what happens.

Ugh. www-gitlab-com is hitting 1GB limit?

mentioned in issue #2935

Ugh. www-gitlab-com is hitting 1GB limit?

Yes, I propose we split the blog out to its own repository. My tests show that the blog currently builds in under a minute: https://gitlab.com/gitlab-com/www-gitlab-com/issues/1681

changed milestone to %WoW

This was fixed with https://gitlab.com/gitlab-com/infrastructure/issues/2935 and it also explains what happened.

I'm closing this one as it was handled.

closed