Configure GitLab's Prometheus server to monitor Kubernetes
Description
Presently as part of 8.17, we are packaging Prometheus and a collection of it's exporters. (Redis, Postgres, Node, and GitLab-Monitor) This allows us to monitor a number of aspects of the GitLab service and the host it is running on.
In 9.0, we are adding our first support for monitoring customer apps with Prometheus as well. This offers the ability to see performance metrics from Prometheus, for apps that are deployed to Kubernetes. (It relies upon two metrics pulled from k8s node's, CPU and Memory)
While we can require customers to bring their own Prometheus server, stand it up in an environment and configure it properly, this is not a trivial amount of effort especially for demo environments like I2P. Since we are already packaging and including Prometheus, it would be great if we could simply enable the monitoring of Kubernetes with the packaged Prometheus server.
Proposal
With GitLab including Prometheus, we should offer an easy way to enable monitoring of not just GitLab, but also Kubernetes. In the short term, we should add a new gitlab.rb flag to enable monitoring of Kubernetes if GitLab is deployed within k8s. While we would like to eventually automatically deploy a Prometheus instance in each Kubernetes cluster, this will allow us to get started much more quickly with little end-user effort.
Since this attribute is about a configuration setting of Prometheus, and not an entire new process to run, I would suggest we simply add this as a sub-option like prometheus['monitor_kubernetes'] with a default value of false.
We should make this configurable as opposed to always on, in the event that someone has a very large cluster and doesn't want this particular server to monitor every single node within it. Otherwise their only alternative would be to totally turn off Prometheus, which is not good.
scrape_configs:
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
metric_relabel_configs:
- source_labels: [pod_name]
target_label: environment
regex: (.+)-.+-.+
replacement: $1