WIP | Separate package POC

Problem

We are shipping more and more new features that depend or build on top of other projects: prometheus, gitlab monitor, kubernetes, etc.

Adding more services increases:

1. Size of the package
2. Effort to maintain the code
3. Required time to install and configure the package
4. Difficulty of providing simple and smooth upgrade paths
5. Chances of something going wrong

Size of the package

In this day and age package size should not be a problem, whether the package is 100MB or 300 MB should not matter. However, it seems that it does as we are getting users that are more vocal about the size of the package. While the amount of people not caring and not being vocal is several magnitudes larger, we should not expect our package to grow more.

Lets put users complaints for a second and think about our infrastructure.

At the moment, the package is around 300MB. Sure, we can trim some of the fat from the package by cleaning up after the build. We have one or two, or three long standing issues to address the package size. It seems though, that the priority of this always gets pushed down the TODO list as it is not as interesting as a new feature.

Now for the numbers:

• We build 20 packages per release.
• According to the numbers for 3 previous months on version.gitlab.com, we;ve released 9 versions in November, 9 in October, 16 in September.
• We have approximately 300k downloads per month.

Lets say we do around 10 releases per month. Each package is pushed to our package server and served from it.

20 packages x 300 MB = 6GB per version

This means we will add 6GB per version to our package server S3.

6GB per version x 10 releases per month = 60GB 

This is 60GB of packages we add every month to our package server that we need to support forever.

According to our AWS production account, only from one S3 bucket we did 13,213.670 GB of traffic for November. If we combine all S3 buckets it takes us up to 15.5 TB of traffic.

Obviously, we have other things in our S3 buckets besides packages but considering we have 300k downloads, it would not be surprising if package size is adding to these numbers significantly.

Effort to maintain the code

Each configuration or feature we add, adds to the complexity of code maintenance. Lets just use an example of PostgreSQL which is not a configuration or a feature but an essential part of our package. Due to us depending on external software, we had great difficulty upgrading GitLab.com to the newer version of Postgres. To support upgrades for our users, we are adding a lot of code. Even with rigorous review, there was an issue that we luckily managed to catch before announcing it for everyone.

Required time to install and configure the package

We currently have 6 services that are not GitLab but are in the package. Two of them are not essential for functioning of GitLab, Registry and Mattermost. Each of these comes with plethora of pitfalls and magic we do in the package to make configuration and install as easy as possible for the users. Each problem has a possibility of preventing user from running their GitLab instance.

Unpacking the package takes considerable time even on our infrastructure. This is costing us a lot of time during GitLab.com deploys. We basically have downtime without even doing the things that actually require downtime. The more files we introduce, the slower unpacking gets. The more configuration we introduce, the slower reconfigure is, the more chances of something going wrong.

Difficulty of providing simple and smooth upgrade paths

Already mentioned with the Postgresql upgrade, but I will use another example; Mattermost. This is a non-essential functionality which carries GitLab name since we ship it. When Mattermost made a breaking change in their upgrade process we had difficulties getting the upgrade process done. We still don't know whether this breaking change caused more issues for our users/customers as we might not have received the info. At least one customer of ours had wanted to not upgrade GitLab because they could not afford to break their Mattermost or use their new functionality. The code we ended up shipped is not optimal and future upgrade might require more hacks.

Proposal

While there is only so much we can do with what we already shipped, we still can do something about the features we want to ship.

First candidate for this is monitoring. This feature is mostly interesting for larger instances.

This is what initial proposal of shipping prometheus is bringing into the package:

 14M	./bin/node_exporter
 29M	./bin/prometheus
 12M	./bin/promtool

This is 55MB in 3 binaries only, more configuration in the cookbook and we are just getting started.

With this in mind, I propose that moving forward we ship all new non-essential features in a separate package, eg. gitlab-extras.

This new package should be separately installed after gitlab-ce/ee. Configuration would remain in /etc/gitlab/gitlab.rb. The extras would be configured using a separate cookbook but there would still be one gitlab-ctl reconfigure.

Pros

• Our main package size will stabilize
• We can add gitlab-ctl command that will fetch the package automatically and install it for users system if they have internet access
• Upgrades can be separate to GitLab which means possible downtime only for extra features and not the whole of GitLab
• It is likely that this package won't require upgrades as often as GitLab so you can grab it when you need additional functionality

Cons

• Upgrades need to be separate (similar to gitlab-runner)
• Uninstallation will be more complicated, do we need to care about the uninstallation order?

POC

I used name gitlab-monitoring for this project. It is still within the same omnibus-gitlab project, no separate repo needed. Software definition is separate from gitlab and it contains all libraries that we need. Since we expect this to work only with Gitlab installed, we don't have to duplicate installed software. The path where this is installed can also be tuned, but for the sake of POC I've namespaced it to /opt/gitlab/monitoring .

The cookbook that controls the binaries is still within the main gitlab package but this can be moved to be within the secondary package.

You install gitlab-ce_8.1.0+git.1763.5a573b2-rc1.ce.0_amd64.deb first and then gitlab-monitoring_8.1.0+git.1763.5a573b2-1_amd64.deb, and run one reconfigure.