Improve alerting in GitLab.com using alert manager
We need to have proper alerting in GitLab.com, improve the messages so they make sense and do paging properly.
As an outcome we also need to:
-
Have clear alerts in the slack channel that point to the proper runbook when an alert arises. -
Clear noisy alerts that are not actionable out of the channel -
Have proper documentation that explains we can add new alerts making it a no brainer. -
Have the alert manager triggering alerts into slack and/or pager duty when it matters.
cc/ @eliran.mesika