Trim and Prune Artifacts
We are currently lugging around 22 TB of artifacts over in:
root@worker-web01:/var/opt/gitlab/gitlab-rails/shared/artifacts# du -khs .
22T .
root@worker-web01:/var/opt/gitlab/gitlab-rails/shared/artifacts#These are all broken down by year and month:
root@worker-web01:/var/opt/gitlab/gitlab-rails/shared/artifacts# ls -alh
total 2.2M
drwx------ 20 git root 4.0K Mar 1 00:06 .
drwxr-x--x 8 git gitlab-www 117 Dec 9 16:59 ..
drwxr-xr-x 13 git git 4.0K Nov 29 2015 2015_11
drwxr-xr-x 93 git git 4.0K Jan 1 2016 2015_12
drwxr-xr-x 207 git git 8.0K Jan 31 2016 2016_01
drwxr-xr-x 451 git git 12K Feb 29 2016 2016_02
drwxr-xr-x 683 git git 20K Mar 31 2016 2016_03
drwxr-xr-x 1850 git git 44K Apr 30 2016 2016_04
drwxr-xr-x 1970 git git 48K Jun 7 2016 2016_05
drwxr-xr-x 2342 git git 56K Jun 30 2016 2016_06
drwxr-xr-x 2929 git git 72K Jul 31 2016 2016_07
drwxr-xr-x 3950 git git 96K Sep 14 13:07 2016_08
drwxr-xr-x 4850 git git 116K Oct 1 10:35 2016_09
drwxr-xr-x 5540 git git 132K Nov 1 01:02 2016_10
drwxr-xr-x 6969 git git 164K Dec 4 20:58 2016_11
drwxr-xr-x 6374 git git 152K Dec 31 23:58 2016_12
drwxr-xr-x 7832 git git 188K Feb 1 19:48 2017_01
drwxr-xr-x 8374 git git 200K Mar 2 19:20 2017_02
drwxr-xr-x 3084 git git 76K Mar 5 03:14 2017_03
drwx------ 4 git git 44 Nov 25 2015 tmpFOUR DAYS into March and we're already at 1.1 TB, the numbers for 2017 so far are as follows:
- March 2017 (4 days) : 1.1 TB
- February 2017 (28 days) : 4.2 TB
- January 2017 (31 days) : 3.7 TB
We need an automated way to depreciate and trim artifacts because they are just going to keep growing out of hand.
Activity
@pcarranza this is what I was talking about the other day. @stanhu, @ayufan and @tmaczukin - we need to address this.
Can we force a prune somehow? 22TB is a lot of data that is resting for nothing.
What about build logs @northrup?
@stanhu It has an option to set default, but we still need to discuss what will be default.
@pcarranza We need to discuss that with Product/Marketing about how to do it as we would affect data stored on GitLab.
@stanhu Yes, we have the most of data in DB.
@ayufan I propose we set the default to 2 weeks.
@markpundsack I think that this is also a part of the bigger problem: data retention for CI: artifacts, logs, pipelines. I think that we need to start a discussion with rest of the team what should be our policy on that. As
2 weeksaffects new, doesn't affect old ones.How should we proceed with enabling that? Do we need the separate announcement (to 9.0), or not?
@ayufan How about we at least announce it with 9.0? If we're really concerned, we could give everyone a couple week grace period before enabling it, so they can update their
.gitlab-ci.ymlto set something longer if they know they need to. And yes, we'll need to clean up older ones at some point soon. Maybe we can announce that at the same time.I wish our expiry was sensitive to leave the last X most recent ones...
-
@ayufan we talked about moving artifacts handling to carrier wave.
Is there an issue about that? and if not, where should I open it?
cc/ @markpundsack
mentioned in issue #1307 (closed)
Artifacts issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/29203.
/cc @joshlambert
Since the default doesn't affect existing artifacts (correct @Kamil?), we do need a way to prune the existing artifacts, perhaps by script.
How is this as a starting point?
- Communicate to end users of GitLab.com that these changes are coming in 1 month. We should also re-send the same communication probably at 1 week left, and then 1 day left.
- Communicate that we are implementing a 2 week expiration by default, unless otherwise specified in
gitlab-ci.yml. They should take action now if they want different behavior, for example setting a specific value for tagged artifacts. - Communicate that we will also be pruning existing artifacts. Artifacts older than 2 weeks will be deleted, unless they are tagged.
Edited by Joshua Lambertmentioned in issue #1455 (closed)
