Add alert for abnormal increase of exclusive locks

When we see this:

I would like to get some form of alert so we are aware, because in the middle of the noise it's hard to tell that we are actually orders of magnitude over the standard exclusive locks rate.

This is one the main symptoms of people not being able to comment on issues - we were deleting a user that was too ingrained in the system because he was marked as spam, and it was failing (luckily) because it took too much time to complete the transaction.

ALERT ...
IF pg_locks_count{...} / avg_over_time(pg_locks_count{...}[1h]) > 5
FOR 5m
...

and

ALERT PostgresTooManyRowExclusiveLocks
  IF pg_locks_count{mode="rowexclusivelock"} > 100
  FOR 5m
...

cc/ @maratkalibek @ahanselka @yorickpeterse @jschatz1

added alert label

Related: https://gitlab.com/gitlab-com/infrastructure/issues/914

mentioned in issue #914 (closed)

mentioned in merge request runbooks!208 (merged)

For the second alert, here is the WIP MR - https://gitlab.com/gitlab-com/runbooks/merge_requests/208

For the first alert conditional expression it is not valid always, because pg_locks_cound can be zero which results in NaN.

Maybe we should rewrite this in following form - pg_locks_count{...} > 5 * avg_over_time(pg_locks_count{...}[1h]). But in this case if there was no locks for one hour and it there is at least one lock, this alert will be triggered.

closed

reopened