Create `gitlab/kitchensink` docker image

Description

Most .gitlab-ci.yml configurations start from a standard base image and then apt get whatever they need to run their CI/CD. This is great and functional, but all that apt geting adds a lot of time to the CI/CD pipeline. We can encourage people to create their own custom images, and that helps, but is quite painful and still inflates download time and the proliferation of images overwhelms our docker image proxy cache. If we provided a standard image with most common tools already installed, we could speed up a lot of pipelines by providing everything that 90%+ of our users need. It would get even better if we pre-loaded this image on every shared runner on GitLab.com. This would let us approach the raw speed of CircleCI or Travis CI.

Proposal

Start from something common like ubuntu and add:

• docker, docker-compose, etc.
• rvm and a bunch of ruby versions
• nvm and a bunch of node versions
• Google Cloud CLI which includeskubectl
• awscli
• dpl (in every ruby version)
• python, java, other languages (and multiple versions if feasible)

Links / references

• Meta speed issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/21624

mentioned in issue #21624 (moved)

I will say in general, that I think this is a fine idea and will work for many people. I will add a few points to consider just to make sure we realize what this is good for and what it is not.

Uber containers sound good when you are first starting a project. And if they help you get started quickly, then that is a reason enough to do them. However, they often fall down as a project grows. For example say I am using node 6.4.2 and jdk 1.8 (I am). And say in 6 months when java 1.9 comes out, I start using that and suddenly need java 1.9 in my build. Two things happen: 1) I now have to wait for the maintainers of the uber container to test and feel comfortable with upgrading to java 1.9 in the uber containers. So I may be waiting. If I get antsy and need it sooner, then I have to move away from the uber container. 2) The new uber container comes out, but it also has node upgraded to 6.7. I am now forced to get node 6.7 to get java 1.9, and oh by the way my stuff isn't compatible with node 6.7 yet. So now I need to do a bunch of extra work.

One reason to do docker based builds is that they follow the unix pipeline mentality. One container with one specific purpose (e.g. build node stuff, build java stuff, build golang stuff, etc.). And since they are so fast to spin up the penalty is minimal. Now yes of course you want to have a local docker repo probably and have some of these images cached so id doesn't take a long time to pull them into the system. But I suspect that the runners will wind up with the images you are using cached locally anyway.

All that said, I am all for the concept and it can be useful to help folks get going.

This seems tempting, but generally I prefer to avoid of building this fat images. It will really fast grow beyond resonable point.

The other problem is by introducing Uber image we would have to be really careful with changes, upgrading / deprecating of versions since people will depend on specific structure and versions present in the images.

The other idea is to have multiple smaller images with different versions focused around different versions of specific language (something similar what is done by Travis). However, this kind of defies a preloading part of the solution, since we would have to preload tens of gigabytes. If we don't preload we will have to download a few gigabytes which effectively slows down the build. Given my experience I still find using Library of Docker Images to be more flexible then using one uber image.

Mentioned in issue #21624 (moved)

mentioned in issue #28552 (moved)

@ayufan Yeah, this is a maintenance nightmare, but something our competitors do, and our customers expect. It also happens to be pretty effective. Luckily the industry is leaning more towards Docker and people controlling their own images, so this is becoming less of a concern. But still, a very real concern.

Re: versions: We can certainly support people using non-latest images, if they run into some problem due to an upgrade. They won't get (all) the speed benefits of a preloaded image though. I mean, the image updates will probably share layers, so they'll still get some benefits, but not all of them.

Crazy idea, but how about just using CircleCI's build image?

• https://github.com/circleci/image-builder#ubuntu-1404-xxl-enterprise (I believe XXL-Enterprise is what we'd want)
• https://hub.docker.com/r/circleci/build-image/tags/
• https://circleci.com/docs/1.0/build-image-trusty/

mentioned in issue #35221 (moved)

mentioned in issue gitlab#6032

moved to gitlab#6033

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

mentioned in issue gitlab#8449

mentioned in issue gitlab#11108