Create standalone registry container for Helm

As part of our larger effort to create per-service container images (https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2420), we wanted to choose Registry as the first service to work on. This is primarily because it is the most loosely coupled, and so would be a good first step in investigating how to configure by itself.

A couple areas we will need to discuss:

1. What configuration was Omnibus performing to set up Registry?
2. Without relying upon Omnibus/chef, at least for the configuration on the Registry side, what should we use to apply and coordinate the settings?

cc @marin and @pcarranza. I wanted to make an issue just to get this started, and hopefully allocate some time within the 9.4 release cycle to dedicate to this on our side. So just put something together as a starting point.

changed milestone to %9.4

Would something like the Container Registry, mailroom, or Mattermost be easier to start? Sidekiq has the same dependencies (e.g. environment variables, config files, etc.) as the Rails/unicorn layer, so it seems a lot trickier to get right. But perhaps since we are doing this on GitLab.com it makes sense to tackle this first.

@stanhu we also discussed Container Registry as a starting point as well, which I would also be fine with. That one I believe was thought to have more configuration requirements to setup the authentication and other aspects, which brings up the important question of what we want to use to manage that in lieu of Omnibus in this model. (Terraform? Kubernetes? etc?)

My inclination is that Sidekiq would provide bigger bang for buck than Container Registry, so that may be a more interesting place to start to get some gains on the .com side.

changed the description

added cloud images feature labels

I don't think this is what we concluded in our meeting @joshlambert. The conclusion was that we will work together with Production to correctly configure our current environment so we understand how things can be separated. Registry is actually the first thing we will look at for configuring. Once we have a better understanding, we will decide which service is a first candidate that we could containerise and use on GitLab.com.

changed the description

Thanks @marin, tweaked the description to reflect Registry as the first case. Please update the description as needed.

mentioned in issue charts/charts.gitlab.io#14

changed title from Create standalone sidekiq container to Create standalone registry container for Helm

@omame and @marin are working together on this one

assigned to @marin

assigned to @omame

mentioned in issue #2420

@omame and myself had a session this morning noting down the work.

We talked about:

1. Current situation on GitLab.com => tl;dr: Registry runs on all nodes except git. This is unnecessary but configuring service separately is hard.
2. Configuration that exists for configuring both the Registry and GitLab rails registry related services inside of omnibus-gitlab => Noting in the followup comment https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2441#note_31923283
3. Next steps we will be taking => Issue in https://gitlab.com/gitlab-com/infrastructure/issues/1993
4. Document

mentioned in issue gitlab-com/infrastructure#1993 (closed)

Current omnibus configuration (The Magic of the package hiding the configuration)

Registry service configuration in config.yml

• registry[‘registry_http_addr’] = “localhost:5000” It has to be specified WITHOUT the scheme. Otherwise registry crashes. This is the address on which the Registry service listens.
• registry[‘http_secret’] 64 char hex => Find in docs what this specifcially does
• registry[‘token_realm’] Auth endpoint => https://gitlab.example.com/jwt/auth
• registry[‘issuer’] => This is being populated from gitlab_rails[‘registry_issuer’]. We might need to have a separate options for registry in order to populate this field in this service.
• registry[‘rootcertbundle’] => Path to the certificate file. This should be by default, path /var/opt/gitlab/registry/gitlab-registry.crt
• registry[‘storage’] is the hash containing the details for storage. By default this has contains the path to the shared directory: {filesystem: { rootdirectory: ‘/var/opt/gitlab/gitlab-rails/shared/registry’}, ‘cache’ : {blobdescriptor: ‘inmemory’}, "delete":{"enabled":true}} . You can specify here S3 creds.

GitLab Rails service for Registry in gitlab.yml

• gitlab_rails[‘registry_host’] This is the host value of the registry external url. In our example, this “registry.example.com”. Does not contain ports or scheme
• gitlab_rails[‘registry_port’] This is the external port. This is the port on which the end user reaches the registry. This is listen port specified in “/var/opt/gitlab/nginx/conf/gitlab-registry.conf”. Lets use the example of gitlab.com AND dev.gitlab.org. On gitlab.com we have registry_external_url ‘https://registry.gitlab.com’ AND we have a SEPARATE certificate for it. On dev.gitlab.org, we are reusing GitLab certificate so registry_external_url ‘https://dev.gitlab.org:5005. This means that we do not have to have a separate certificate to serve registry. In the case of gitlab.com, the value of ‘registry_port’ is going to be empty (because we use 443 by default). In the case of dev.gitlab.org, the value of registry_port` is going to be 5005.
• gitlab_rails[‘registry_api_url’] = “http://localhost:5000” This is the registry endpoint WITH the scheme. This is the address on which the registry is running.
• gitlab_rails[‘registry_path’] = ‘/var/opt/gitlab/gitlab-rails/shared/registry’ its the storage where GitLab will store images, by default.
• gitlab_rails[‘registry_key_path’] => This is the path to certificate key. It is the pair with “registry[rootcertbundle’]” = “/var/opt/gitlab/gitlab-rails/etc/certificate.key”
• gitlab_rails[‘registry_issuer’] Gets added to the registry’s config.yml. It is the signer of the certificate. This is to prevent certificate issue spoofing: we tell registry who is authorised to issue certs. We configure this to by default “omnibus-gitlab-issuer”. You shouldn’t care about changing this in most cases anyway. The only important thing about this is that it has to be the same config. value in registry config.yml

Additional settings interesting for Production team:

• registry['internal_certificate']: This is the actual certificate string. This will be saved at registry[‘rootcertbundle’] location.
• registry['internal_key']: This is the key (as a string). This ends up in /var/opt/gitlab/gitlab-rails/etc/certificate.key. This path ends up in rails, not in registry. Don’t get mislead by the option name. The reason why it is named this way is because it is a certificate-key pair.

added Cloud Native and removed cloud images labels

Update in https://gitlab.com/gitlab-com/infrastructure/issues/1993#note_33881344

This is (hopefully) the final version of how the registry works at GitLab.

For the record, registry is now running as an isolated service on GitLab.com.

added Scheduled label

changed milestone to %10.0

assigned to @WarheadsSE and unassigned @marin

In working through this with helm, I've faced a few questions, which I will cover below, but have decided that for the time being, I will simply start working with each sub-chart having an ingress for itself. So the chart that I am finalizing for helm.gitlab.io now, will include

• registry container (in ReplicaSet, default of 1 replica)
• service (to expose the containers)
• ingress (to expose the service to the outside of the cluster/minikube)

Things I have been facing are both simple and complex, but I'll summarize them as briefly as possible.

How do we share secrets across components that might be in different namespaces/clusters/locations. There are several things that need to be shared to coordinate all of the components that make up a complete GitLab application stack. This could be accomplished with a Consul service or the ilk.

How do we want to prepare some of these secrets? As it stands, I have pulled the registry related information from my own GitLab VM to be able to get the pods to connect, but many other components need information to either be pre-made or generated and stored. One such set of items is the SSL key pairs that are used for communication with the Docker Registry for authentication via GitLab's provided JWT.

How do we layout ingresses? Do we create one per service, or some mix of individual and separated services that use a shared ingress? Can we auto-generate this information somehow?

marked this issue as related to #2420

A follow-up on the above:

I have implemented the registry container via helm sub-chart and am working on the documentation to it now.

I have laid everything out as I had described above for the sub-chart, and have gated the dedicated Ingress behind a boolean flag ingress.enabled so that you can either have it as a part of the 'global' Ingress, or a dedicated one.

In terms of secrets, I have been working under the presumption that everything will currently be deployed either into the same namespace or the necessary secrets will be populated into the correct namespaces at the time of deployment.

For the sake of the RSA/SSL keypair used by GitLab (gitlab-rails) and the Registry, I have made us of a configurable secret for the public side of the pair that is passed into the Registry container. This way, even if the Registry and GitLab are on different namespaces/clusters, we'll still be able to have them communicate. This does mean that you'll need to generated them outside of GitLab and have the private side of the pair passed into those containers.

@omame Could you have a look at https://gitlab.com/charts/helm.gitlab.io/merge_requests/2 ?

The chart has been merged at this time. Currently, it makes use of Kubernetes Secrets for the TLS & Token certificates.

assigned to @joshlambert and unassigned @WarheadsSE and @omame

assigned to @WarheadsSE and unassigned @joshlambert

closed