Add high error rate alert (!334) · Merge requests · GitLab.com / runbooks · GitLab

Do not update/delete: Banner broadcast message test data

Do not update/delete: Notification broadcast message test data

username-removed-274314 requested to merge pc-web-error-rate-alert into master Aug 24, 2017

https://prometheus.gitlab.com/graph?g0.range_input=1h&g0.expr=sum(backend_code%3Ahaproxy_server_http_responses_total%3Airate1m%7Btier%3D%22lb%22%2C+environment%3D%22prd%22%2C+backend%3D%22web%22%2C+code%3D%225xx%22%7D)+-+sum(backend_code%3Ahaproxy_server_http_responses_total%3Airate1m%7Btier%3D%22lb%22%2C+environment%3D%22prd%22%2C+backend%3D%22web%22%2C+code!%3D%225xx%22%7D)&g0.tab=0&g1.range_input=1h&g1.expr=sum(backend_code%3Ahaproxy_server_http_responses_total%3Airate1m%7Btier%3D%22lb%22%2C+environment%3D%22prd%22%2C+backend%3D%22web%22%2C+code%3D%225xx%22%7D)+-+sum(backend_code%3Ahaproxy_server_http_responses_total%3Airate1m%7Btier%3D%22lb%22%2C+environment%3D%22prd%22%2C+backend%3D%22web%22%2C+code!%3D%225xx%22%7D)&g1.tab=1

This is how this works ☝

It's quite aggressive, one page after 15 seconds of high error rate (higher than everything else together)