Expose build performance data to Prometheus

Currently we run build and forget about it. For Kubernetes and Docker we can read CPU/Memory/IO/Network usage over time and have very detailed build performance metrics.

We should expose them with Prometheus endpoint.

The data send to prometheus should be: per-project, per-used-machine, per-shortened-runner-token, per-stage (in which is build now).

Since we will have this data, we should also push summary (maximum, average, std-dev) of the performance to GitLab where we could show and quickly access that if needed. Maybe another way would be to print that at the end of build log with information about average CPU usage, maximum memory usage, IOPS and IO Read/Write, network Read/Write.

added Deliverable label

No sure about performing the agg at that point, maybe you could get some input from @bjk-gitlab on this.

Kudos for the effort, it will be a massive help.

Aggregate would only be printed on client-side to build log. As a something for runners that would not use Prometheus.

Everything else: potential graphs in GitLab would be done similarly to current Metrics integration.

mentioned in issue gitlab-com/infrastructure#1416

Generally, no, we don't like to have aggregation points like this. We should have Prometheus talk to Docker (cAdvisor) or Kubernetes (Kubelet) directly about container stats.

It depends on as what you see as container state is an outcome of how runner works, it can be useful for less detailed info, but generally runner knows much more about it state as it can divide metrics into steps allowing you to know exactly when each of the command is executed, like cloning.

This could also be probably achieved with timestamps.

added To Do label

Maybe another way would be to print that at the end of build log with information about average CPU usage, maximum memory usage, IOPS and IO Read/Write, network Read/Write.

This would influence the failed pipeline email and display this data here, not the actual failure. I think we should just push the data to prometheus. (or let it pull)

changed milestone to %Backlog

@markpundsack @sytses @tmaczukin

This is a very interesting idea, but we decided to go Boring Way and do the simplest first iteration: use existing tools to monitor runners: cAdvisor and Node Exporter running on auto-scaled machines, hooked to auto-discovery and being scrapped by Prometheus.

That would allow us to:

1. Move forward with https://gitlab.com/gitlab-com/infrastructure/issues/1666,
2. Figure out what we need in terms of monitoring of runner jobs,

The proposed approach of building that into runner was also raised by @bjk-gitlab as not ideal and not Prometheus-way.

To implement out-of-bound monitoring of Runner Jobs, and due to auto-scaled and ephemeral machines, we need some way to discover all nodes to be scrapped. GitLab Runner can easily interact with every Docker Engine (in the case of Kubernetes easily read metrics out of Pods), but cannot easily expose that information outside. Since we decided to do the simplest iteration we need a way to auto-discover all machines running. This can be achieved with the usage of service discovery, ex. Consul. Any created machine will connect to Consul and advertise itself. Prometheus will then monitor Consult to get a dynamic list of machines to monitor.

So basically Consul can be considered as one of external dependencies that allow us to move forward, but this will not be built into product it would not force to use that approach later as we will be able to understand what are our requirements in terms of monitoring (possibly relying scrapping Kubernetes completely) and choose better approach what we want to achieve.

Since this is using existing components/tools we will document how we achieved that, allowing anyone else to replicate what we have done and have some preliminary way of monitoring the CI jobs.