Add an endpoint for proxying artifacts on a GitLab server

Per https://gitlab.com/gitlab-org/gitlab-ce/issues/34102#note_38326228

We want to be able to serve any file in any GitLab artifact from the Pages daemon.

Pages will need to take a new option: -artifacts-server

Authentication is out of scope for this iteration, meaning only public artifacts will be browsable in this way.

Some representation of the full path of the project (to support subgroups) will be joined with the job ID to create a unique domain label. One suggestion is group~group~group~project~jobID. Another is artifact~<projectID>~<jobID>. This will then be joined to pages-domain to create an unambiguous endpoint.

• Make sure all possible combinations are globally unique
• What about label length? We're limited to 63 characters, maximum nesting is 20...

Requests to that endpoint will cause Pages to hit the .../raw endpoint in GitLab and stream the content to the user, setting both Content-Type and Content-Length correctly.

• Range queries and other oddities?
• Caching? I assume we'll want basic Cache-Control headers just like the rest of Pages.

We've been talking elsewhere about making Pages operate on artifacts directly, rather than having the current -pages-root and the unpack step in gitlab-ce: https://gitlab.com/gitlab-org/gitlab-pages/issues/68#note_35039355 . This will be a good chance to investigate how such a solution could look without breaking existing setups!

@ayufan are we hooked on the artifact identifier being human-readable? Since job IDs are already globally unique (I think), maybe we're better off just using job-<id>.artifacts.gitlab.io ?

I wonder if we can use existing pages domain for that. Can we use some character, as separator that will not be a valid project path?

The subdomain doesn't have to be human readable. I think that it may be better if this is actually unique and encrypted / obfuscated.

cc @briann

@ayufan we can have the default configuration be artifacts.<pages-domain> but I like giving people the option to disable it or move it to another domain entirely.

Ach, projects and groups can be called artifacts. Rubbish.

-artifacts is fine for now.

Due to wildcard SSL certificate we cannot use double subdomains :)

True, and we don't want to be handling -artifacts-domain-ssl stuff as well.

OK, let's hardcode it to artifact~<job-id>.<pages-domain> and do away with -artifacts-domain. That should be globally unique, coverable with the existing SSL certs, generatable in gitlab-ce, and won't conflict with any namespaces.

Phew!

mentioned in issue #68

marked the checklist item Make sure all possible combinations are globally unique as completed

marked the checklist item What about label length? We're limited to 63 characters, maximum nesting is 20... as completed

added In dev label

The other implication of using the bare job ID is that we'd need some endpoint in gitlab-ce that can relate it to the project for us.

It might be easier to use artifact~<project-id>~<job-id>. We can then use http://docs.gitlab.com/ce/api/jobs.html#get-job-artifacts to get the file data.

We might need to add an endpoint to grab a specific file from the archive, but that's definitely possible.

It would require access token in order to interact with API.

@ayufan not necessarily - we already allow public projects to be accessed via the API, and there's an MR in your queue that adds that ability for groups too. Publicly available artifacts is an easy addition.

When we come to add private content support, we're already talking about OAuth support, so Pages could use that token to access the content.

Interesting. We would need to allow accessing individual files via artifacts. It seems easy.

WIP taking shape in https://gitlab.com/gitlab-org/gitlab-pages/merge_requests/42/diffs

I'm torn between allowing individual file downloads or just using the existing "download full zip file" endpoint.

How did you find that ZIP wrapping library? Was it performant? Does the GitLab API support the request headers it needs?

mentioned in merge request !42 (closed)

changed the description

@nick.thomas Content ranges will not work on top of .zip archive. Is it performant? Quite performant, but keep in mind that it does use at least 4 requests to S3 to fetch the file as we have to read .zip header and then seek over to read the file.

So, it is good to be used for lightweight usage. Otherwise, the penalty will be quite high and we might need to store an archive in extracted form.

Let's add a new endpoint to GitLab CE then, per gitlab-ce#37196

Yes I agree, this should be a caching responsibility of GitLab or Workhorse.

mentioned in merge request !44 (merged)

closed via merge request !44 (merged)