Monitor and/or automate cleanup of load balancers and persistent disks
When we hit our limit of load balancers on GCP in gitlab-internal
the ce:kubernetes
and ee:kubernetes
builds will fail since they need to create more of these.
We can add some monitoring to notify in #g_configure
when we have reached our load balancer limit. This will save us time when debugging why CI is failing and perhaps even allow us to fix it before CI breaks.
See slack conversation about how to add the monitoring.
Also further to this there may be some way we can automate the cleanup of unused (orphaned) load balancers (perhaps a scheduled task or a chatbot command). See the manual process we use to cleanup load balancers now.
-
Manual instructions for cleanup are described in https://gitlab.com/gitlab-org/gitlab-development-kit/issues/384.
-
Command line instructions are described below:
gcloud compute disks list --filter="-users:* name:gke-qa-cluster-* zone:us-central1-a" | tail -n +2 |cut -d' ' -f1 |xargs -n1 gcloud compute disks delete --quiet --zone us-central1-a