Too many targets down in Prometheus

We currently have 216 targets down in Prometheus:

Looking at the targets list it seems like most of them are either returning 404, 500 or connection refused.

We should revive those targets.

changed the description

@omame good catch. We want the monitoring systems relevant, and not just ignored because it produces too much noise.

First batch identified and solved.

Gitaly isn't running on the front-end anymore but no one told Prometheus. I just did and now we're down to 167.

Other causes for this include unicorn scraping, influxdb exporter and /process on port 4567, but I haven't investigated too much.

Here's the list of jobs with targets down, with a handy counter.

assigned to @omame

changed milestone to %WoW

I just spent some time reviewing the prometheus targets and I took down the number of failed ones to 51. Most of what's left is elasticsearch, which @victorcete is dealing with, and some Chef leftovers that @jtevnan is cleaning up right now.

Besides that we should only have a handful of targets left to fix but I think that falls down into regular administration.

While I was at it I've also updated the targets graph in the fleet overview to represent the percentage of available targets and I've set up thresholds to colour the value if it falls below a certain value.

closed

I've removed gitlabcom-elasticsearch targets (those were 16 nodes)