Create/manage Kubernetes cluster on GKE for GitLab group

/cc @tauriedavis @ayufan @sytses

For now, I just want to mockup that show a deep integration with Kubernetes and GKE. I'm less worried about technical complexity.

Some future considerations:

Support multiple clusters, each matched to an environment name or wildcard (e.g. for review apps), so projects in the group would automatically put their production apps in the production cluster, for example.
Support clusters created for projects, not just groups.
Clusters tab

mentioned in issue #27890 (moved)

Current dialog when GKE creates a cluster, which we can greatly simplify:

mentioned in issue #27889 (moved)

There are also a lot more options if you click "More" above the create button. For this mockup, we are only showing an option for the number of nodes correct? We are removing the rest of these options for now for simplicity.

@tauriedavis yeah, just number of nodes. You don't even need to name the cluster. We'll pick the machine size and everything else.

mentioned in issue #27893 (moved)

This flow closely matches what we are doing with mattermost. You would be able to create a new cluster either when creating a new group or in the group settings.

The success message includes a link to the kubernetes dashboard. The dashboard link, along with the subdomain and IP, are also generated when you enable a cluster (is this possible? would this be possible on the new group page before the group is created?).

Once you enable a cluster, you have the option to add the number of nodes and this can always be updated later.

New group

| Unchecked | Checked | Success message | |---|---|---|---|---| | | | |

Edit group

Unchecked	Checked	Success

Thoughts @markpundsack @dimitrieh @sytses?

I'll look closer in the morning, but that's an awesome start!

Should it be named GKE or just Kubernetes, as Container Engine doesn't say whether this is Docker / Swarm / Kubernetes.

I like it. The only thing about technical feasibility are the group-level services.

I updated the header in the comps above to include (GKE).

It would also be pretty easy to transfer this flow to project creation/edit if we decide to do that.

Yeah, we don't currently have group-level services, but we sorely need them, and especially for something like this, I'd expect companies to create a single clusters for all their apps to share, at least when they're getting started. So while it's tolerable to have a Kubernetes service at the project level where you can just copy-paste the same data to all related projects, that doesn't work when actually creating clusters. At any rate, this is a vision, not something that has to be implementable immediately.

For the dashboard, subdomain, and IP, those seem like they'd be better shown in a separate Cluster settings page, especially because you probably can't or at least don't want to set up all those things until you actually hit save, at which point you're no longer looking at that page.

(Maybe call it Container Cluster for added clarity`)

Also, another reason for having it as a separate page is that people should be able to add their own custom domain(s) to the cluster.

Looking at this, I then think we should have a link to that Cluster settings page from this area where you enable/disable the cluster. Then of course, that leads me to think maybe for a project, we should have links to the related settings pages near every place where we enable/disable those settings. But that's a story for another time.

@markpundsack

especially because you probably can't or at least don't want to set up all those things until you actually hit save, at which point you're no longer looking at that page.

This is why I also included a dashboard link in the success message. But I was hoping that the links could be generated automatically - it looks like that is what we are doing with mattermost (according to the mockups).

Also, another reason for having it as a separate page is that people should be able to add their own custom domain(s) to the cluster.

Is this something we plan to do later or something we want to show now? I think it would be okay to add this configuration in the same spot.

Looking at this, I then think we should have a link to that Cluster settings page from this area where you enable/disable the cluster.

Like we talked about yesterday, I'm not sure we need a cluster settings page right now if we start with just a single cluster. If they hit "enable" we could automatically generate the links. Changing the number of nodes would be reflected in the kubernetes dashboard. All said, I do agree that we will need a "manage" button if we have the enable button on the main settings tab and a separate cluster tab for everything else (which isn't much right now). I'm just not sure it makes sense to create a new tab for one cluster if it is possible to generate the links immediately.

But I was hoping that the links could be generated automatically - it looks like that is what we are doing with mattermost (according to the mockups).

@tauriedavis This is different than Mattermost in that we wouldn't want to reserve an IP address until after they have hit save. We can give the subdomain, but that's fully under GitLab control, but the IP is not (unless we pre-reserved a bunch of them or something). But at any rate, it just doesn't make sense to provide that information until after you hit save.

Is this something we plan to do later or something we want to show now?

I think it's fine for now; it gets the point across.

What do you think about not generating the links on clicking enable, but after save, but still keeping it all on the single page?

Considering that we are adding graphs for https://gitlab.com/gitlab-org/gitlab-ce/issues/27890, I created a new flow with the cluster on it's own tab. I feel strongly that the place where you enable the cluster should also be where you display cluster information and other (future) configuration options. With this flow, you can still enable the cluster when creating a new group, but the enable option will be on the cluster tab in settings, rather than general.

New group

| Unchecked | Checked | Success message | |---|---|---|---|---| | | | |

Edit group

Unchecked	Success	Checked

cc @markpundsack

@tauriedavis Hmm... isn't that contrary to how we do project settings? There's a project settings page where you can enable/disable (and with EE set visibility) for repos, issues, snippets, wiki, CI/CD Pipelines, and container registry, but then have separate pages to configure each of those. Maybe we don't do that for groups, but that's the mental model I was working from when suggesting it.

@tauriedavis At any rate, this is awesome and will do nicely. We can argue about location later. :)

To nitpick:

the cluster size should probably default to 3.
Dashboard should say "Kubernetes dashboard: https://104.154.60.172/ui" to make it clear that it's the k8s dashboard we're talking about, and unless we do something special, there's just IP access to it.
IP should just be an IP like "Static IP: 243.233.322.32".
Typo in "Your contain cluster". I'd drop "above" as well. It reads funny to me.

P.S. Now that I think about it, we could add DNS routing to their dashboard, under something like k8s.<group>.gitlab-apps.com or <group>.gitlab-apps-admin.com.

P.P.S. For the technical watcher, the k8s dashboard is protected by basic auth. We could either put the username and password in the link itself so when you click on it, it's pre-filled. Or we add a "Show credentials" link like GKE does. I vote for simplicity, but there's probably a reason GKE doesn't encourage people to connect directly. Probably because you don't want to show the creds by default in case someone is watching over your shoulder or you're recording a video call. So my next thought was to have it in the link, but not the text on the page, but then I realized when you hover over the link, it's common to show the destination URL in the status bar at the bottom of the browser window, which would end up linking the creds anyway. And then even if we hide that behind Javascript, when you go to the URL, you might see the creds in the URL bar (at least temporarily). We could potentially proxy the traffic through GitLab, and thus handle permissions ourselves, perhaps with OAuth.

I don't think the project settings should be copied if it means a worse experience. In fact, I think the current settings needs a lot of help UX-wise. ;)

I did try really hard to make placing the enable portion in the General section but every iteration hid the generated links from the user. Given that we only are offering one configuration option, it felt necessary that a user should be aware of the dashboard link and see them being generated after they hit save. They shouldn't be hidden on another tab or down the page, out of view. I also thought about duplicating them in the success message at the top, but this could lead to confusion as to where to find them later. With the cluster having it's own section, a user can hit save and easily see what is being generated. But yes, still open for arguing about location later. @awhildy Would love your thoughts on location - spent a long time thinking through putting it on general page but couldn't get the flow right.

Updated mockups based on nitpicks - also added them to the issue description.

New group

Unchecked	Checked	Success message

Edit group

Unchecked	Success	Checked

I don't think the project settings should be copied if it means a worse experience. In fact, I think the current settings needs a lot of help UX-wise. ;)

Yeah, that's fair. And of course you know consistency matters too, but we can apply new UX philosophies to the group settings and back-port to project settings, if need be. However, I happen to think having a single place to enable/disable a bunch of project features is actually a good thing, with optional/advanced configurations of those features relegated to a different page. And the current EE settings made that a lot more complex/uglier because of the visibility drop-downs. At any rate, this is off topic, so I'll drop it.

@tauriedavis This all looks great! No further changes from me.

@sytses what do you think?

@tauriedavis - Agreed on the settings perspective. There is currently too much going on in general settings. Once we can improve upon it in #22210 (closed), we might have a more consistent, cleaner model. But for now, it makes sense to keep it as it's own tab. It would feel odd to split it currently with just the one option.

changed title from Kubernetes settings to Create Kubernetes cluster on GKE

changed title from Create Kubernetes cluster on GKE to Create/manage Kubernetes cluster on GKE for GitLab group

mentioned in issue #28351 (moved)

Related: to ease onboarding, perhaps we could leverage Google's free trial offering. I think you can do a 3-node cluster for a month or so for free. We could make that really easy to enable and encourage. Going further, it would be awesome if we could leverage that same kind of offer without having people log in through Google at all. Imagine they create a new group or project and have an option of automatically creating a new cluster, and it's as simple as a single click, without charge to them or GitLab.

this is just a quick sumup of my mind regardings kubernetes:

it's nice that you think a lot about easier deployment, cloud integration etc.
please keep in mind, that it is important (for whatever reasons), that services used with gitlab can be self hosted
self hosting kubernetes is rather complex, however minikubes is on a good way.

self hosting things like a heroku compatible version of dokku[1] is very simple at the opposite

so maybe it makes sense to describe the whole process? Will kubernetes integration be bound to google at some point in the future or always be self hostable?

[1] https://github.com/dokku/dokku

mentioned in issue #31218 (moved)

mentioned in issue #32638 (moved)

@kaystrobach Kubernetes integration will not be bound to Google, but this issue is specifically about how we could make it easier to create a cloud-hosted Kubernetes cluster, via Google GKE. If you have your own cluster, you can already do that today with the Kubernetes service integration.

mentioned in issue #35616 (moved)

changed the description

marked this issue as related to #35616 (moved)

Thinking about this more, I think we're going to need to support multiple clusters since it's against best practice in many companies to use a production cluster for dev/test. Also, this really should be at the group level. I see it best fitting with https://gitlab.com/gitlab-org/gitlab-ce/issues/35616.

mentioned in issue #35892

mentioned in issue #35954 (closed)

mentioned in issue #35956

We don't yet have group-level Kubernetes integrations, so cutting scope, we could add a cluster creation button to the existing project-level k8s service. (https://gitlab.com/gitlab-org/gitlab-ce/issues/35954)

We already have Google login with OAuth, so we'll need to figure out how to escalate the granted permissions to include cluster creation.

Then we'll need to decide if we can get away without installing Helm, Prometheus, etc. automatically, or if that's required for an MVP. My guess is that it is. The initial iteration may exclude it, but to make it "viable", we'll need that part automated as well. Technically, we might be able to get away without installing runners as the GitLab instance could have shared runners already (e.g. GitLab.com), but the cluster needs Prometheus and ingress for Auto Deploy/Auto DevOps to work, and those are best installed with Helm. There's a separate issue to install Prometheus with a click. Perhaps we should rely on that. Maybe extend it with a single click to install ingress. At least we should work with Prometheus to coordinate. /cc @joshlambert

Lastly, we'll need to expose or automatically handle ingress IP. If we expose it, we should include instructions for configuring your DNS. I wonder if we should put this at the k8s service level, or leave it as an independent project variable. Auto Deploy requires it so it feels like a critical part of the MVP.

MVP proposal moved to: https://gitlab.com/gitlab-org/gitlab-ce/issues/35956

mentioned in issue #28916 (moved)

Thanks @markpundsack on the Prometheus side. The issue tracking deployment of Prometheus in Kubernetes is: https://gitlab.com/gitlab-org/gitlab-ce/issues/28916

I think we can hopefully lead the charge here on a few items:

Communication between resources outside of cluster (say GitLab.com) and those inside of cluster using k8s proxy API
Deployment of a resource from the GitLab UI into Kubernetes using a Helm chart
Some of the UI work as well

marked this issue as related to #35956

I wonder if we should consider this simple checkbox style cluster creation for new projects too. I believe people will prefer creating/managing clusters at the group level, but at least for testing, people will probably start small, and create a new cluster for a new project (or add to an existing project) rather than creating a new group, or adding a cluster to an entire existing group. Also, if we only do it at the group level, how will people do this for personal accounts?

changed the description

This is no longer relevant as we're going a different direction with a dedicated Clusters page inside groups and projects. Extracting the important ideas from the proposal and closing.

Also, since we know we need more than one cluster per group or even per project, creating a cluster at group creation time is less obviously important. Maybe someday we'd have a group creation option to create two clusters, one for production and one for dev/test/qa. Or whatever the best practices turn out to be.

Automatically create subdomain <group>.gitlab-apps.com: https://gitlab.com/gitlab-org/gitlab-ce/issues/38369
Monitor Kubernetes cluster: https://gitlab.com/gitlab-org/gitlab-ee/issues/3537
Cluster configuration/management: https://gitlab.com/gitlab-org/gitlab-ee/issues/3538

closed

Create/manage Kubernetes cluster on GKE for GitLab group

Description

Proposal

Mockups

Links / references

Designs

Child items ...

Activity

Admin message

Admin message

Create/manage Kubernetes cluster on GKE for GitLab group

Description

Proposal

Mockups

Links / references

Relates to

Activity