Kubernetes/GKE integration

Description

We want to make integration with Kubernetes and GKE really simple for users, so they can leverage a cluster for many things (review apps, deployments, runners, monitoring, etc...).

Proposal

10.1

• Create a Cluster page to add a new cluster on GKE: https://gitlab.com/gitlab-org/gitlab-ce/issues/35954

10.2

• Easy installation of helper applications in Kubernetes cluster: https://gitlab.com/gitlab-org/gitlab-ce/issues/38464
• Move Kubernetes from service to Clusters page: https://gitlab.com/gitlab-org/gitlab-ce/issues/35616

10.3

• Support existing clusters in GKE
• Group-level Cluster page: https://gitlab.com/gitlab-org/gitlab-ce/issues/34758

GA (10.4-10.6)

• Allow selection of existing GKE clusters in Cluster page: https://gitlab.com/gitlab-org/gitlab-ce/issues/38786
• Multiple cluster support: https://gitlab.com/gitlab-org/gitlab-ce/issues/27322
• Single Click Deploy to GCP: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2437

Future

• Automatically create subdomain <group>.gitlab-apps.com: https://gitlab.com/gitlab-org/gitlab-ce/issues/38369
• Monitor Kubernetes cluster: https://gitlab.com/gitlab-org/gitlab-ee/issues/3537
• Cluster configuration/management: https://gitlab.com/gitlab-org/gitlab-ee/issues/3538

mentioned in issue #28916 (moved)

marked this issue as related to #28916 (moved)

mentioned in issue #35958

marked this issue as related to #35958

marked this issue as related to #35954 (closed)

changed the description

mentioned in issue #27888 (closed)

/cc @joshlambert @bikebilly @eliran.mesika

The reason I am thinking of buttons to install things rather than checkmarks during creation, is then it works whether your cluster was created by us or not. One solution, solves 2 use-cases. Decouples creation from configuration.

And even for the common case, it's only 4 easy button presses.

@dimitrieh @tauriedavis @pedroms Would love your thoughts on this flow.

More backstory in https://gitlab.com/gitlab-org/gitlab-ce/issues/27888#note_36729007 and https://gitlab.slack.com/archives/C433ZLXSB/p1501794524669057.

It also partially solves https://gitlab.com/gitlab-org/gitlab-ce/issues/27889, although from a different angle.

mentioned in issue #32831 (moved)

If for whatever reason we decide to install Prometheus etc. using k8s API directly, rather than Helm, we should consider making option to install Helm. Currently, Auto Deploy install helms itself to the project's namespace so it wouldn't be the end of the world if we didn't have tiller there, but it feels weird.

@markpundsack we should be consistent unless there is a really good reason not to. The helm chart looks like it has most of the features, so don't expect there to be a problem.

changed the description

changed the description

marked this issue as related to #35616 (moved)

marked this issue as related to #27888 (closed)

marked this issue as related to #27322 (moved)

marked this issue as related to omnibus-gitlab#2437

changed the description

@ayufan How feasible does this plan look to you?

changed title from MVP of GKE integration to GKE integration

removed milestone

Changed the description to include MVP, 1.0, and post-MVP plans. Removed milestone from this issue.

changed the description

changed the description

Since this connects with https://gitlab.com/gitlab-org/gitlab-ce/issues/28916 that is supposed to happen earlier than MVP we should coordinate the efforts with Prometheus Team cc @joshlambert.

I would advise that we store this information in an easily queryable form that can easily be accessed by performing DB query. The reason for doing that is to have an ability to:

• implement usage pings,
• perform automated upgrades if needed if a new security release is pushed,
• propose easy to use the system that will be used also for tracking Helm install, Runner installs, Nginx installs, etc.

Maybe we are looking at storing this data in the separate table, like: kubernetes_applications that would keep track Helm-installed applications.

Totally agree @ayufan we should stay aligned here, and I think your suggestions make a lot of sense. It's a Stretch for us in 10.0, but !

changed the description

changed the description

Added https://gitlab.com/gitlab-org/gitlab-ce/issues/36629 for Helm Tiller.

changed the description

mentioned in issue #32638 (moved)

@bikebilly This issue needs updating with our current plans.

@bikebilly #34758 (moved) should be about group level Clusters page, no? Investing in the service integrations seems a dead end.

@bikebilly We also need new issues for moving from service integration to clusters page for manual cluster creds. Maybe a separate issue for connecting to an existing GKE cluster.

Edit: Nevermind, I see they're in 10.3 now. We just need an issue for one of them.

changed the description

@bikebilly Can you create an issue for "Support existing clusters in GKE"?

changed the description