Prometheus doc updates for 9.4

doc/ci/environments.md

@@ -602,9 +602,8 @@ exist, you should see something like:
 >**Notes:**
 >
 - For the monitor dashboard to appear, you need to:
-  - Have enabled the [Kubernetes integration][kube]
-  - Have your app deployed on Kubernetes
   - Have enabled the [Prometheus integration][prom]
+  - Configured Prometheus to collect at least one [supported metric](prometheus_library/metrics.md)
 - With GitLab 9.2, all deployments to an environment are shown directly on the
   monitoring dashboard

doc/user/project/integrations/prometheus.md

@@ -23,11 +23,11 @@ Integration with Prometheus requires the following:
 ## Getting started with Prometheus monitoring
-Depending on your deployment and where you have located your Prometheus server, there are a few options to get started with Prometheus monitoring.
+Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring.
 * If both GitLab and your applications are installed in the same Kubernetes cluster, you can leveraged the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes).
 * If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
-* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-proemtheus-outside-of-kubernetes).
+* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes).
 ### Getting started with Prometheus outside of Kubernetes
@@ -69,7 +69,7 @@ kubectl apply -f path/to/prometheus.yml
 Once deployed, you should see the Prometheus service, deployment, and
 pod start within the `prometheus` namespace. The server will begin to collect
 metrics from each Kubernetes Node in the cluster, based on the configuration
-provided in the template.
+provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod).
 Since GitLab is not running within Kubernetes, the template provides external
 network access via a `NodePort` running on `30090`. This method allows access
@@ -139,8 +139,9 @@ environment which has had a successful deployment.
 > [Introduced][ce-10408] in GitLab 9.2.
 > GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages.
-Developers can view the performance impact of their changes within the merge
+> Requires [Kubernetes](prometheus_library/kubernetes.md) metrics
+Developers can view theperformance impact of their changes within the merge
 request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot
 indicates when the current changes were deployed, with up to 30 minutes of
 performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after

doc/user/project/integrations/prometheus_library/kubernetes.md

 # Monitoring Kubernetes
 > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
-GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor).
+GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor). No additional services or exporters are needed.
 ## Metrics supported
@@ -12,8 +12,15 @@ GitLab has support for automatically detecting and monitoring Kubernetes metrics
 ## Configuring Prometheus to monitor for Kubernetes node metrics
-Prometheus has internal support for discovering and monitoring of Kubernetes node metrics.
-If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
-To configure your own Prometheus server 
+In order for Prometheus to collect Kubernetes metrics, you first must have a
+Prometheus server up and running. You have two options here:
+- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
+- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes).
+## Specifying the Environment label
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
+
+If you are using [GitLab Auto-Deploy][autodeploy] and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added.

doc/user/project/integrations/prometheus_library/metrics.md

 # Prometheus Metrics library
 > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
-GitLab offers
-## Metrics and Labels
-GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
-and `container_memory_usage_bytes`. These metrics are collected from the
-Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
-container or Pod running in the cluster.
-In order to isolate and only display relevant metrics for a given environment
-however, GitLab needs a method to detect which pods are associated. To do that,
-GitLab will specifically request metrics that have an `environment` tag that
+GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
+* [Kubernetes](kubernetes.md)
+* [NGINX](nginx.md)
+* [Amazon Cloud Watch](cloudwatch.md)
+We have tried to surface the most important metrics for each exporter, and will be continuing to add support for additional exporters in future releases. If you would like to add support for other official exporters, [contributions](#adding-to-the-library) are welcome.
+## Identifying Environments
+GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
+
+In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
+GitLab will look for the required metrics which have a label that
 matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
-If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
-configuring Prometheus above, the `environment` will be automatically added.
-## Configuring Prometheus to collect automatically collected metrics within Kubernetes
-In order for Prometheus to collect Kubernetes metrics, you first must have a
-Prometheus server up and running. You have two options here:
-- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the
-  [bundled version of Prometheus][promgldocs]. In that case, follow the info in the
-  [Omnibus GitLab section](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes)
-  below.
-- If you are using GitLab.com or installed GitLab outside of Kubernetes, you
-  will likely need to run a Prometheus server within the Kubernetes cluster.
-  Once installed, the easiest way to monitor Kubernetes is to simply use
-  Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
-  In that case, follow the instructions on
-  [configuring your own Prometheus server within Kubernetes](../prometheus.md#configuring-your-own-prometheus-server-within-kubernetes).
+For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
+## Adding to the library
+We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `additional_metrics.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml) file.
+> Note: The library is only for monitoring public, common, system services which all customers can benefit from. Support for monitoring [customer proprietary metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/2273) will be added in an subsequent release.

doc/user/project/integrations/samples/prometheus.yml

@@ -24,6 +24,44 @@ data:
           target_label: environment
           regex: (.+)-.+-.+
           replacement: $1
+      - job_name: kubernetes-pods
+        tls_config:
+          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+          insecure_skip_verify: true
+        bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
+        kubernetes_sd_configs:
+        - role: pod
+          api_server: https://kubernetes.default.svc:443
+          tls_config:
+            ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+          bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
+        relabel_configs:
+        - source_labels:
+          - __meta_kubernetes_pod_annotation_prometheus_io_scrape
+          action: keep
+          regex: 'true'
+        - source_labels:
+          - __meta_kubernetes_pod_annotation_prometheus_io_path
+          action: replace
+          target_label: __metrics_path__
+          regex: "(.+)"
+        - source_labels:
+          - __address__
+          - __meta_kubernetes_pod_annotation_prometheus_io_port
+          action: replace
+          regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
+          replacement: "$1:$2"
+          target_label: __address__
+        - action: labelmap
+          regex: __meta_kubernetes_pod_label_(.+)
+        - source_labels:
+          - __meta_kubernetes_namespace
+          action: replace
+          target_label: kubernetes_namespace
+        - source_labels:
+          - __meta_kubernetes_pod_name
+          action: replace
+          target_label: kubernetes_pod_name
 ---
 apiVersion: v1
 kind: Service