Investigate possibility of supporting LFS content stored in object storage

Description

We should consider moving LFS content off of local/NFS into object storage. This is for two primary reasons:

1. Object storage is linearly scalable, whereas NFS and local storage are more difficult to do so.
2. Object storage is cheaper than NFS or local storage.
3. It is also easy to make geographically redundant.

For example consider cost for GitLab.com, and our competitors pricing:

• BitBucket charges $0.10/GB, and does not seem to meter Bandwidth. ($5 for 50GB of storage)
• GitHub charges $0.10/GB of combined bandwidth and storage. ($5 for 50GB of storage and bandwidth)

Considering AWS for example, there are 3 main storage types (assuming GitLab server is hosted in AWS):

1. S3 - $.023/gb
2. EBS - $.025/gb (cold HDD) to $.1 (SSD)
3. EFS - $.3/gb

When comparing Cold HDD to S3 be aware of the difference in durability, as well as S3 being reachable from any GitLab worker node. (Also options of S3-IA, etc.)

Proposal

When setting up GitLab, we could have a server level configuration option which could take S3 (or other object provider, although many are S3 compatible) credential information. Once verified connectivity is there, we could then use that in the future for storing LFS data. This could also use the same settings as the CI counterpart, with a checkbox to save either CI or LFS (or both) in that account..

One challenge is working with user authentication. Git LFS primarily uses HTTP Basic Auth, which S3 for example doesn't support. One option is to simply have GitLab proxy this data on the backend. If GitLab is hosted in the same cloud provider as the object storage, typically this additional hop of bandwidth is free.

We should also consider migration, as this may be a nice feature for larger orgs who have been testing on CE or EES and need to migrate data. Having a productized solution would also help GitLab.com migrate, as well. For example, we may have a flag to indicate where a particular artifact is (local vs. object) which we can update as the migration proceeds. Once complete, the local admin can deal with the existing artifacts as needed. (Delete, archive, etc.)

@mydigitalself your thoughts since this deals with Git Repos?

@chriscool I think what you're working on (basically integrating LFS into Git itself, but in a backend-agnostic way) could allow the use S3 as an external object database, right?

added gitlab-ce~129601 label

@rymai yes there is no restriction regarding the backend in what I am working on. It will be necessary to implement the S3 credential authentication though.

It looks like the following could already help:

http://www.s3auth.com/ https://github.com/yegor256/s3auth

@chriscool Awesome, thanks!

changed milestone to gitlab-ce%29

@joshlambert certainly something to consider for the future, i've put it on the backlog, thanks for the mention.

Currently, GitHub uses S3 as storage for LFS. They do that by sending pre signed upload URL, as you cannot pass to the client the basic auth. This is the same approach that Runner uses to upload files to S3-like storage, Workhorse uses to upload artifacts to S3-like storage.

mentioned in issue #2097 (closed)

changed milestone to %9.5

Adding SFDC link to prospect who requested something similar in https://gitlab.com/gitlab-org/gitlab-ee/issues/2097

https://na34.salesforce.com/00161000006fyqb

Distributed object storage support: I'm not sure this one matters much for small scale installations. But for us, we put a lot of work into scalability and HA that would have been saved if the repository stores could have been stored outside of local filesystem and partitioned.

@DouweM this is, in essence, a duplicate of https://gitlab.com/gitlab-org/gitlab-ce/issues/13825 right?

@mydigitalself Yes

Only question would be whether this should be in CE or EE.

I'm going to close this one as a dupe and move the original one into EE.

closed