Support multiple Kubernetes cluster per project
Description
It's common to have a different Kubernetes cluster for production vs the rest of development. Currently, the project-level integration with Kubernetes only supports a single Kubernetes cluster per project. We should consider allowing multiple clusters, or at least a different cluster for production.
Proposal
Links / references
Activity
mentioned in issue #27521 (closed)
I'm surprised this wasn't in the initial design of the integration. Based on the documentation for Kubernetes, it makes more sense to have the configuration the opposite from how it's setup.
I wrote up a stackoverflow post to confirm this is how the integration was setup, was sad to see I was correct. http://stackoverflow.com/questions/43302601/gitlab-9-x-kubernetes-integration
Any idea on how difficult of a change this would be? I would love to integrate my deployments with Gitlab's Deploy Board but need this change before I can.
Proposal:
If Kubernetes Integration is enabled on the project level, allow users to override these settings per each environment. I.e. append exactly the same configuration mask to the environment edit page (and later possibly to pipelines YAML). Also the Kubernetes Integration configuration mask would need to be extended with a new field for the "cluster name".
Such solution should provide enough flexibility for everyone.
I would also extend the ticket description with the "support of multiple namespaces per cluster", because this is equally important.
Thanks for your feedback @ivanmrva. We like to decouple issues, so multiple namespaces should be a separate issue with cross links between them.
We now have a pretty good proposal for environment-specific variables (https://gitlab.com/gitlab-org/gitlab-ee/issues/2302) based on an environment
scopewith wildcards. We could do something similar with services, if we let you have multiple services, each with their own scope. It seems pretty significant to extend all services this way, and quite possibly unnecessary. So perhaps the single Kubernetes service can handle this internally./cc @bikebilly @ayufan
@msvechla You could theoretically use multi project pipelines as a workaround for this, as you can configure a sep cluster for each project.
@WarheadsSE however warns about being careful
Edited by Dimitrie Hoekstra@dimitrieh that would be a hack and we would end up with a lot of duplication. What needs to be done to push this? We basically can not make any use of the nice deployment / ui features without being able to define clusters per environment (or multiple clusters per project at all).
@markpundsack @bikebilly one other data point we received today, was that we should also consider multiple clusters for a single environment as well. The reason being is when you have something like production running in multiple regions, you are going to have multiple k8s clusters.
This is likely going to be typical for larger k8s production deploys.
mentioned in issue #35956
we should also consider multiple clusters for a single environment as well.
@joshlambert I'm not sure about this. Semantically, I assume you mean, there could be multiple clusters for production, but we use the word environment to mean a unique deployment target, thus each of those regions would likely be their own environment.
Specifically, we only support one environment per job, so if you're deploying to multiple places, you'd have to use multiple jobs. If you're using multiple jobs, you can use multiple environment declarations. e.g.
production/euandproduction/us. In the UI, those will be grouped underproduction, but will be distinct environments. Then for k8s creds, we'd do something similar to what we do for variables, and let you specifyproduction/*to share creds with all production environments, orproduction/euif you need a specific one for the European cluster.I don't think we need multiple clusters per environment per se.
marked this issue as related to #35616 (moved)
marked this issue as related to #35956
Yea, I haven't spent significant time about applications running in multiple datacenter and whether they should be a single or multiple set of environments. Grouping and using multiple makes sense, although a few thoughts come to mind there:
- Coordination between the two environments would seem to get more tricky, for example:
- Would you want to run a single canary in one cluster/region, scale that slowly out to a full cluster/region, and only then if that is running well continue on to more clusters/regions? Or would we recommend starting a canary deploy in each cluster in parallel? Seems like the best plan would be canary in a single cluster first, as that would be lower risk.
- It may change some of the UI design, when a folder could mean a group of disparate and unrelated environments but also a single environment spread out across multiple regions. For example, detailed monitoring dashboards across a group may not make a lot of sense for
review/*but it would be critical forproduction/* - I do like the informational model though it could provide, as you'd inherently get the ability to zoom in/out as needed into a particular region/cluster or zoom out to the whole.
- Coordination between the two environments would seem to get more tricky, for example:
Would you want to run a single canary in one cluster/region, scale that slowly out to a full cluster/region, and only then if that is running well continue on to more clusters/regions? Or would we recommend starting a canary deploy in each cluster in parallel? Seems like the best plan would be canary in a single cluster first, as that would be lower risk.
@joshlambert The good news is that we can support whatever flow they like. But yeah, we should think about what best practices to recommend. I'd think canary into single cluster would make most sense. Then just do incremental rollout to the other clusters; there's very little value in doing canaries to every region.
It may change some of the UI design, when a folder could mean a group of disparate and unrelated environments but also a single environment spread out across multiple regions. For example, detailed monitoring dashboards across a group may not make a lot of sense for
review/*but it would be critical forproduction/*Yeah, that's an excellent point. We don't have any rollup monitoring yet, but we probably should if this is a common and important pattern.
As a side note, I think people are hoping the future contains a single k8s cluster that includes multiple regions that magically distributes apps across regions and heals itself when there are problems. So if you're using multiple regions just for Geo/HA reasons, that might be abstracted out at the cluster level. But having said that, some companies have different content for the same app in different regions, so we still need to support this. Also, people have other use cases, like an admin-only instance of the app.
mentioned in issue #35616 (moved)
Labelled customer+ because of: https://gitlab.slack.com/archives/C3UCHUA76/p1503924975000235 (internal)
Edited by Zeger-Jan van de Wegadded customer+ label
changed milestone to %Next 3-6 months
This is something we'll hopefully solve with https://gitlab.com/gitlab-org/gitlab-ce/issues/35616 in 10.3, but if not, hopefully soon thereafter.
Yes, per @zj ^ we have a current EES customer (https://na34.salesforce.com/0066100000HPKpP) who is looking at an enterprise strategy with K8s and and GitLab but this will be a very important element for that path forward.
moved to gitlab#8007
Hey!
GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current
gitlab-cerepository will become a read-only mirror, without any proprietary code. All development is moved to the currentgitlab-eerepository, which we will rename to justgitlabin the coming weeks. As part of this migration, issues will be moved to the currentgitlab-eeproject.If you have any questions about all of this, please ask them in our dedicated FAQ issue.
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855You can also find more information here:
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304- https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/
Frequently Asked Questions
For an up to date list of questions and answers, please take a look at
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855What will we do with the repository names?
https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.
Why rename gitlab-ce to gitlab-foss?
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear
I created a merge requests for CE, and this got closed. What do I need to do?
You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.
How does the licensing work in this new setup?
Everything in the
ee/directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in theee/directory, the process of contributing changes is the same as when using the gitlab-ce repository.Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?
Yes, these changes will be synced to gitlab-foss automatically, several times per day.
