From 2434d8cd5d3f52b78b731643b96ded70bf788a6f Mon Sep 17 00:00:00 2001
From: Marat Kalibekov <marat@kalibek.kz>
Date: Wed, 5 Apr 2017 14:58:54 +0600
Subject: [PATCH 1/2] Add information about available channels for alertmanager
---
howto/alerts_manual.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/howto/alerts_manual.md b/howto/alerts_manual.md
index 4d8f9e9..95f2c9a 100644
--- a/howto/alerts_manual.md
+++ b/howto/alerts_manual.md
@@ -23,7 +23,7 @@ The common procedure is as follows:
ALERT runners_cache_is_down
IF probe_success{job="runners-cache", instance="localhost:9100"} == 0
FOR 10s
- LABELS {severity="critical", channel="infrastructure", pager="pagerduty"}
+ LABELS {severity="critical", channel="production", pager="pagerduty"}
ANNOTATIONS {
title="Runners cache has been down for the past 10 seconds",
runbook="howto/howto/manage-cehpfs.md"
@@ -31,7 +31,7 @@ ALERT runners_cache_is_down
}
```
-This will result in a critical alert posted in slack channes `#prometheus-alerts` and `#infrastructure`, pagerduty with a link to https://dev.gitlab.com/cookbooks/runbooks/blob/master/howto/manage-cehpfs.md. Important part is the end or url - `howto/manage-cehpfs.md`. It is taken from annotation `runbook`. Runbook will provide information how to manage situation alerted. Main principle of the runbook should be - `don't make me think`.
+This will result in a critical alert posted in slack channes `#prometheus-alerts` and `#production`, pagerduty with a link to https://dev.gitlab.com/cookbooks/runbooks/blob/master/howto/manage-cehpfs.md. Important part is the end or url - `howto/manage-cehpfs.md`. It is taken from annotation `runbook`. Runbook will provide information how to manage situation alerted. Main principle of the runbook should be - `don't make me think`. For channel you can use `#production`, `#ci`, `#gitaly` values.
### What if I want to add more data?
--
GitLab
From 5ccb3a4e83281cf6e50c8092f88b9fb38c363937 Mon Sep 17 00:00:00 2001
From: Marat Kalibekov <marat@kalibek.kz>
Date: Wed, 5 Apr 2017 15:07:38 +0600
Subject: [PATCH 2/2] How to add more slack channels to alertmanager
---
howto/alerts_manual.md | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/howto/alerts_manual.md b/howto/alerts_manual.md
index 95f2c9a..ccfc34e 100644
--- a/howto/alerts_manual.md
+++ b/howto/alerts_manual.md
@@ -70,6 +70,30 @@ All alerts are routed to slack and additionally can be paged to PagerDuty.
Currently we are not using email alerting rules.
+### Add more slack channels to alerts
+
+1. In routes section of [alertmanager.yml template](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/blob/master/templates/default/alertmanager.yml.erb) add following:
+```
+ - match:
+ channel: gitaly
+ receiver: slack_gitaly
+ continue: true
+```
+1. In receivers section [alertmanager.yml template](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/blob/master/templates/default/alertmanager.yml.erb) add following. Note that `send_resolved`, `icon_emoji`, etc values must be taken from `slack_production` receiver:
+```
+- name: slack_gitaly
+ slack_configs:
+ - api_url: '<%= @conf['slack']['api_url'] %>'
+ channel: '#gitaly'
+ send_resolved: true
+ icon_emoji: ...
+ title: ...
+ title_link: ...
+ text: ...
+ fallback: ...
+```
+
+
### Note about alerts which not fit in any routes
1. Alerts which are routed by default route will be sent to `#prometheus-alerts` channel in slack.
--
GitLab