[Meta] Project Storage Management
Description
For users of very large GitLab instances, for example GitLab.com, managing storage space can become a challenge. For example on GitLab.com there is a 10GB repository limit, but no limitation on storage consumed by CI for artifacts, build logs, or containers.
Large enterprise installations would like to more closely manage this, and in fact some community tools have emerged for managing CI artifacts and container images. Further in the case of CI, some users may not be aware of how much space they are consuming depending on the default settings.
A method for easily viewing the amount of storage consumed, and managing it would be very helpful.
Proposal
GitLab should make it transparent to see how much storage is being consumed, by what types of files, and allow easy management. In the case of large installations, project size limitations would help to reduce management overhead.
To accomplish this, we will need to build a few new features:
- https://gitlab.com/gitlab-org/gitlab-ce/issues/30300 - Project storage to include all storage types
- #30336 (moved) - Provide an easy method to view and manage the storage space consumed at a Project level
- #30411 (moved) - Provide an easy method to view the storage space consumed at a Group & Subgroup level
Enterprise Edition Premium features:
- #30422 (moved) - Support for limiting personal account storage space
- #30421 (moved) - Support for limiting group storage space
Designs
Relates to
- gitlab #284110.07
Activity
added feature proposal label
Related issues:
- gitlab-com/infrastructure#1500
- gitlab-com/infrastructure#1455 (closed)
- gitlab-com/infrastructure#1498
Edited by Ernst van Nierop@joshlambert is this also the place to address the idea of "bring your own S3"?
@ernstvn I am opening a separate ticket, with it linking back to #30336 (moved). I am still doing some research on this, but will update that ticket (and this one) once fully written up.
You can certainly configure object storage manually in your own CI deployment templates for pushing artifacts, but we could make it pretty transparent I think. The question then is if we make the UX at parity, why buy extra storage from GitLab.com if they can just get it at cost from Amazon. Also if you look at competitors who support this, like Travis, it can end up being pretty complicated: https://docs.travis-ci.com/user/deployment/s3/
That said, for GitLab EE customers (with larger storage needs) this could be a big help to better scale storage at lower cost.
Edited by Joshua Lambert@ernstvn In thinking about "bring your own S3" more, I think we should prioritize https://gitlab.com/gitlab-org/gitlab-ce/issues/30391 instead. We haven't had (as far as I know) significant demand at the project/group level to bring your own storage, and with us selling additional storage for GitLab.com we already have a solution.
By starting here, this would allow us to drive down the cost of our own storage allowing more competitive pricing. It would also help our EEP customers who may have similar needs when they attain a certain critical mass.
We can then schedule this at the project level based on customer demand.
Edited by Joshua Lambertmentioned in issue gitlab-com/www-gitlab-com#1465 (closed)
Soon, Collab for Unity will announce charging prices with a 2gb max before charging fees rumored greater than even you and GitHub combined. Once they do, I'll need to switch ASAP and so will others. Gitlab is not first place, however, because your storage limit is only 10gb. Bringing AWS in (either connecting our own or paying for your additional hosting) would change worlds for me.
There needs to be a way to break 10gb. This is the only thing holding GL back, IMO. I was looking at enterprise features and none of them mattered to me if the limit is still 10gb, for example. In fact, I don't personally need any additional features except greater storage (Doh)!
"why not self hosted?" I don't have time to learn and setup right now. I'd like to migrate later, but simplicity is key, for the time being.
GL rocks and I hope to use GL instead of "others" :)
Edited by username-removed-1240212added Platform label
Will there be a similar option to buy additional pipeline minutes if you exceed your monthly allotment and you are already on the Gold plan?
@jdrumtra we aren't currently planning on doing this right now, that conversation is a on the back-burner, in particular as it is possible to bring your own runners.
With storage, I'm not sure bolting-on storage is the correct approach, it may be better to tier it in a similar way to how we do CI minutes. We could look to improve this in the future with bolt-ons.
https://gitlab.com/gitlab-com/customers-gitlab-com/issues/169
changed milestone to %Next 3-6 months
marked this issue as related to #26897 (moved)
marked this issue as related to #30336 (moved)
marked this issue as related to #30411 (moved)
marked this issue as related to #30336 (moved)
marked this issue as related to #30422 (moved)
marked this issue as related to #30421 (moved)
FYI @sitschner this is the longer term thinking around storage visibility and management.
moved to gitlab#9055
Hey!
GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current
gitlab-cerepository will become a read-only mirror, without any proprietary code. All development is moved to the currentgitlab-eerepository, which we will rename to justgitlabin the coming weeks. As part of this migration, issues will be moved to the currentgitlab-eeproject.If you have any questions about all of this, please ask them in our dedicated FAQ issue.
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855You can also find more information here:
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304- https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/
Frequently Asked Questions
For an up to date list of questions and answers, please take a look at
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855What will we do with the repository names?
https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.
Why rename gitlab-ce to gitlab-foss?
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear
I created a merge requests for CE, and this got closed. What do I need to do?
You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.
How does the licensing work in this new setup?
Everything in the
ee/directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in theee/directory, the process of contributing changes is the same as when using the gitlab-ce repository.Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?
Yes, these changes will be synced to gitlab-foss automatically, several times per day.
