Add Kubernetes pod scraping to Prometheus configuration

When Kubernetes monitoring has been enabled for Prometheus, we should also monitor Pods. We do not do this currently, only Nodes.

This is pretty trivial, for example adding Pod scraping is as simple as adding:

- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod
    api_server: https://kubernetes.default.svc:443
    tls_config:
      ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
    bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

• Add pod monitoring

assigned to @bjk-gitlab

@bjk-gitlab is this one you could work on quick? I thought we actually had this but it turns out it's only Node for now. Should be a simply config file change. If not, I can work on it for 9.4 too, let me know.

changed the description

Yes, this should be easy to do.

added Deliverable label

One note, the service monitoring upstream example requires having the blackbox_exporter. It shouldn't be difficult to bundle this as well, but it's a bit more work than just updating the config.

changed the description

mentioned in merge request !1717 (merged)

changed the description

I've created a separate issue to track service monitoring.

https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2559

changed the description

closed