Collect metrics from runners with mtail

To go with our new docs for CI, we need to start gathering stats from the runners via mtail (https://gitlab.com/gitlab-com/runbooks/merge_requests/214#note_26048284)

mentioned in merge request runbooks!214 (merged)

So, thinking about this a bit more. It would be less effort if we added the Prometheus library to the runner code.

closed

Looking more closely, there is already a Prometheus client in the ci runner code. We can update the metrics there.

@ahanselka just being curious here. What is mtail and why were we considering it instead of Prometheus?

@bjk-gitlab Glad this can be done with Prometheus!

@ernstvn mtail is a program that watches log files and turns information within those logs into Prometheus data. It is a companion to prometheus. In this case there is already a prometheus exporter within the CI runner, and it would be easier to expand the metrics that it gives out instead of using a separate program to parse the logs. This way we would be getting the data directly from the source (CI Runner) instead of having to get it from the logs.

@ahanselka Thanks for the explanation.