Reduce disk IO and space usage during backups

This is based on improvements made to the GitLab CI 8.0 backup script.

• Avoid creating many small intermediate files while backing up builds and uploads by using tar and light gzip compression
• Use same backup/restore code for uploads and builds
• Only store a compressed intermediate DB dump

Added 1 commit:

• 90ddf140 - Reduce disk IO during SQL backup

Added 1 commit:

• 7b71727c - Remove old "files" tarball explicitly

Added 2 commits:

• e7896447 - Keep old path: db/database.sql.gz
• 7d58489f - Remove unused variable

Milestone changed to 8.1

Reassigned to @ayufan

Title changed from WIP Backup improvements to Reduce disk IO and space usage during backups

Added 1 commit:

• 901f5445 - Remove superfluous mkdir -p

Reassigned to @jacobvosmaer

backup tests are broken

Added 1 commit:

• 852526e0 - Spec fixes for new backup contents

Added 1 commit:

• b847baf8 - One more backup spec fix

Reassigned to @ayufan

@ayufan the build is green now.

@jacobvosmaer LG, but there's one quirk:

The code assumes that user always sets builds_path to end with builds/. It will brake if the will be different, like: /var/opt/store/gitlab/gitlab-ci-builds.

I would suggest to pass name to Files class.

mentioned in merge request !1584 (merged)

Yes you are right we need to pass a 'name' argument.

I pushed a new commit @ayufan

Added 1 commit:

• 58260a03 - Do no rely on basename of builds, uploads

@ayufan can we merge this?

mentioned in commit edab429c