Speed up builds on omnibus build machines

@rspeicher noticed that every omnibus build requires recompiling the dependencies all over again.

A quick win might just be to install ccache: https://debian-administration.org/article/129/Speeding_up_recompilation_with_ccache

@stanhu Not sure how ccache will make a difference when we are destroying the Docker container after the build is done?

cc @marin @ayufan

Oh, I didn't realize we were creating every build in a Docker container. Could we mount the cache dir from the host?

We are using autoscaled builders. We are currently uploading some cache to S3 but that is helping just a bit. We need to find a way to speed this up, the builds are slower now than with dedicated machines...

@marin Previously we were using pre-built dependencies?

@ayufan Previously omnibus was detecting whether it already built the software definition and if the cache is not "dirtied" it would just skip the build. I remembered that omnibus used to have s3 caching, I will look into that quickly i case we can have a quick win that way.

@marin If you need any help, ping me. I will be happy to debug and make it faster :)

Managed to cut it down by 20 minutes but it is still long:

A lot of this time is spent on pulling from S3, pushing to S3. I will try to change the bucket location to be closer to the builder machines, that might shave off a couple of minutes.

Reassigned to @marin

Changed title: Install ccache on omnibus build machines → Speed up builds on omnibus build machines

mentioned in issue #1372 (closed)

Summary of the work done with the package builds:

• We created docker package builder images https://gitlab.com/gitlab-org/gitlab-omnibus-builder/merge_requests/12
• We created cookbook to setup the infrastructure for auto-scale runner https://gitlab.com/gitlab-org/cookbook-gitlab-runner/
• We enabled autoscaling for package builders in DO -> Packages build time went up to 75 minutes, RPI builds were not working
• More work was done to add caching to the builds https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/845
• Build time went down to 45-60 minutes with cache
• Fixed problems with RPI builds https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/855
• Enabled autoscaling for RPI package builds in Scaleway -> Builds went down from 9+ hours to 4 hours and we were no longer limited to 2 builders
• We started storing external dependencies in our S3 bucket which means that builds are less susceptible to external issues
• More tweaks have been done which included clearing out the cache and starting building cache from scratch -> This brought build time down to approx 35 minutes with warm cache. PI builds still took around 3.5 hours
• We further improved by making sure that our cache is not dirtied https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/888 -> This brought build time down to 15-20 minutes with warm cache without pushing to packagecloud. RPI builds went down from 3.5 hours to 1 hour 20 minutes

Further improvements:

• Split package building from pushing to packagecloud and make sure we don't upload to s3 when pushing to packagecloud https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1414
• Improve docker image nightly builds https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1413
• Make cache smarter #1472

Related issues have been created, no further action here so closing.

Status changed to closed