Monitor Sidekiq queues and restarts

Today #358 made it apparent that we need real-time monitoring of each of the Sidekiq queue depths. Ideal case is 0 for each of the queues (as see in the Sidekiq admin panel on https://gitlab.com/admin/sidekiq/queues):

In addition, we also need to track the number of MemoryKiller restarts per day.

We could wait for gitlab-org/gitlab-ce#7171, but the Sidekiq API already makes it easy to retrieve this information.

Also we need to be on the look out for any Sidekiq process that says "stopping" but has not changed in a long time.

Added important monitoring labels

Milestone changed to %Milestone-3

Reassigned to @maratkalibek

Interesting metrics on grafana for sidekiq:

I wonder to which machine install this check. Installing it to all worker machines will give us a lot of duplicate alerts during fail. Think installing on one will be enough. Start alerting when total becomes more than 30 or any of them becomes more than 10. However, these numbers not final yet.

/cc @pcarranza @stanhu

Don't alert for now, lets just monitor, you could start installing in just the blessed worker. @maratkalibek

Yeah, one machine should be enough for now. We can add redundancy later if we need it. Wonder if it makes sense to install this on the same machine as the Redis server.

It depends on how are we getting those metrics, and if we have a sidekiq available in the redis machine, which I'm guessing no.

https://dev.gitlab.org/cookbooks/gitlab-checkmk/merge_requests/9

running on blessed worker.

@stanhu @pcarranza tried to run script on redis server, but it took a lot of time to run and without luck, even on worker machine it doesn't give results immediately.

Awesome job! thanks @maratkalibek

Thanks @maratkalibek. For my quick reference, these are the links that seem to work for me:

• https://checkmk.gitlap.com/gitlab/check_mk/index.py?start_url=%2Fgitlab%2Fcheck_mk%2Fview.py%3Fview_name%3Dservice%26service%3DSIDEKIQ_QUEUES%26host%3Dworker1.cluster.gitlab.com

• https://checkmk.gitlap.com/gitlab/check_mk/index.py?start_url=%2Fgitlab%2Fcheck_mk%2Fview.py%3Fview_name%3Dallservices%26service_regex%3DSidekiq%26site%3D

Status changed to closed