Create a chart with images per service

Going forward with the charts, we should consider moving away from the Docker all in one image that we officially support. Having one service per container would allow us to have more flexibility at GitLab.com scale, and better align with cloud native best practices. This would also allow us to cover a section of the market where our omnibus-gitlab package or Docker all in one were not possible to use, due to requiring root permissions.

We should consider creating separate docker images for:

1. Unicorn Web
2. Unicorn API
3. Sidekiq
4. Gitaly
5. Workhorse
6. GitLab Shell
7. Container registry
8. Pages

We should consider using official images for:

1. Nginx
2. Registry
3. Mattermost
4. Runner
5. Prometheus

The docker images we would create would need to have a common way of configuration, probably using ENV variables. The details of this are to be decided.

This would allow us to not only create charts that are simpler to scale, but also allow more flexibility with external services.

These images could still be built in one place together with the rest of our build process.

Separating services would also allow us to remove the root requirement that we currently have in our official image.

To make things clear, we would not be separating the existing Docker image but we would be introducing the 3rd option. This 3rd option would be available through Helm Charts at first, down the line we could add other options.

/cc @joshlambert @markpundsack @twk3

This summary looks accurate to me

Made some minor tweaks, but LGTM! I will also be opening an MR to document this direction in the handbook, and will link it here once it is ready for review.

@marin Really glad to see you thinking about this and I fully support this direction!

Related: Cloud native installation: https://gitlab.com/gitlab-org/gitlab-ce/issues/29407

I'd like to understand the downsides. How much additional overhead is there in maintaining a third option?

As a matter of iteration, we could probably start by using separate images for Nginx, Mattermost, and Prometheus.

I had assumed our registry would require a custom image because of our integration with GitLab user permissions, and our multi-level image support.

@twk3 had suggested we might want to use a different Postgres image that is production-ready, whether we can use someone else's or create our own.

To be complete, perhaps we should add GitLab Runner to the list of custom images.

Can GitLab Pages be a separate service? Not that I specifically care, but it's a separate codebase and daemon, right?

How much additional overhead is there in maintaining a third option?

Like with any new project, initial overhead is not big but as time passes we get hit with maintenance cost. For the initial part, I expect us to spend time, in order of required time:

• Streamlining the release infrastructure
• getting the configuration right between all separate services
• Getting any necessary improvements into GitLab projects

As a matter of iteration, we could probably start by using separate images for Nginx, Mattermost, and Prometheus.

If we want to continue with the existing project, I guess that would work. I wonder though whether it makes more sense to start fresh and start from the "most difficult" part, getting GitLab components containerised?

I had assumed our registry would require a custom image because of our integration with GitLab user permissions, and our multi-level image support.

I am not sure if we will have to use a custom image. If I remember correctly, the only limitation we have is the shared directory.

use a different Postgres image that is production-ready, whether we can use someone else's or create our own.

Ideally, we would want community supported one that we don't have to maintain. In reality though, I suspect we will have to have our own. If nothing else then for iteration speed.

To be complete, perhaps we should add GitLab Runner to the list of custom images.

Sure, I'll add that to the list.

Can GitLab Pages be a separate service? Not that I specifically care, but it's a separate codebase and daemon, right?

Yes it can be. We have similar limitation there as with Registry, daemon is separate but it does require access to the shared directory.

/cc @sytses

I added Container registry and Pages to the separate images.

mentioned in issue gitlab-com/infrastructure#1920 (closed)

Note for tracking purposes, our direction page, and scheduling I have created an issue specifically for the Docker images themselves over in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2420.

changed the description

I added GitLab Shell as a potential image split

I'm quite excited about this project. The tile of the issue is a chart and the current discussion is focused on Docker images. What is the intention for this chart? Multiple containers to a pod communicating over localhost? A pod for each Docker image? Can we push in the direction of MSA, even if not all of the services are microservices?

Proposed: a Chart with charts per service.

Each Service Chart would be for one Service. These services might be kind of big and all-in-one but they can be further decomposed independently and as needed. The GitLab Chart would compose the Service Charts rather than have multiple pods. This has quite a few advantages over a single chart of multiple pods such as:

• Sets the stage for more agility via MSA.
• We can have multiple GitLab Charts - maybe mine uses a service mesh, maybe another does something special with ingress... they all use the same Service Charts.
• Dependency management with the service chart config of the Dockerize services. When chart has a new release, Helm outdated will let me know, but how are new Docker versions published to the package manager?
• Independant SDLC even with the one main Helm Chart - If I bump the version of the Mattermost Chart and deploy, only the Mattermost service is updated, the rest of the services my main chart depends on stay put.
• Service Charts can be more generalized for broader use/contrib.

@tom.davidson glad you are excited about this project, we are too! The intent is as you mention, creation of a docker image per GitLab service. We will then augment our Helm charts to make the deployment and on-going management of this larger set of services easy to use.

We have more documentation available here, which provides a broader picture of our objectives, although we are still iterating on some of the naming and design at this early stage so your feedback is welcome and appreciated.

As a first step towards this goal, we are looking to separate out the GitLab Registry service into its own container. The work is taking place in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2441.

mentioned in issue gitlab-org/omnibus-gitlab#2466 (closed)

marked this issue as related to gitlab-org/omnibus-gitlab#2420