Add runbooks for redis and rack attack bans

README.md

@@ -114,6 +114,11 @@ The aim of this project is to have a quick guide of what to do when an emergency
 * [Create VMs in Azure, add disks, etc](https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/azure.md#managing-vms-in-azure)
 * [Bootstrap a new VM](https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/new-vps.md)
+### Manually ban an IP or netblock
+
+* [Ban a single IP using Redis and Rack Attack](howto/ban-an-IP-with-redis.md)
+* [Ban a netblock using HAProxy](howto/ban-netblocks-with-haproxy.md)
+
 ### Debug and monitor
 * [Tracing the source of an expensive query](howto/tracing-app-db-queries.md)

howto/ban-an-IP-with-redis.md

+# Blocking individual IPs using Redis and Rack Attack
+
+## First and foremost
+
+* *Don't Panic*
+* Be very careful with Redis. There are commands that can be run from the command line that lock it up entirely without warning.
+
+## Background
+
+Redis is the session store and short-term cache. It's used by the Rack Attack module to perform temporary bans and throttling
+against specific IP addresses. It should not be used to ban netblocks.
+
+## How do I
+
+### Connect to Redis
+
+Connecting to Redis is done via the `redis-cli` command line program included with Omnibus. Replace `10.0.0.1`
+with the IP of a Redis node.
+
+```
+worker$ /opt/gitlab/embedded/bin/redis-cli -h 10.0.0.1
+10.0.0.1:6379> auth <password>
+10.0.0.1:6379>
+```
+
+### Find the Redis master node
+
+Writes to Redis can only be performed from the Redis master node. To determine which
+host is the master node you can query the currently connected node:
+
+```
+worker$ /opt/gitlab/embedded/bin/redis-cli -h 10.0.0.1
+10.0.0.1:6379> auth <password>
+10.0.0.1:6379> info replication
+# Replication
+role:slave
+master_host:10.0.0.2
+master_port:6379
+master_link_status:up
+master_last_io_seconds_ago:0
+master_sync_in_progress:0
+slave_repl_offset:xxx
+slave_priority:100
+slave_read_only:1
+connected_slaves:0
+master_repl_offset:xxx
+repl_backlog_active:0
+repl_backlog_size:xxx
+repl_backlog_first_byte_offset:xxx
+repl_backlog_histlen:xxx
+```
+
+Or if you're already connected to the master:
+
+```
+# Replication
+role:master
+connected_slaves:4
+slave0:ip=xxx,port=6379,state=online,offset=xxx,lag=1
+slave1:ip=xxx,port=6379,state=online,offset=xxx,lag=1
+slave2:ip=xxx,port=6379,state=online,offset=xxx,lag=1
+slave3:ip=xxx,port=6379,state=online,offset=xxx,lag=1
+master_repl_offset:xxx
+repl_backlog_active:1
+repl_backlog_size:xxx
+repl_backlog_first_byte_offset:xxx
+repl_backlog_histlen:xxx
+```
+
+Note that the Redis master node can move _while you are connected_ to it. So you
+may find that you get an error when writing a new key saying you cannot write to
+a slave node. When this happens repeat the above process to find the new master.
+
+Redis-cli will also drop authentication frequently, forcing you to re-auth.
+
+### Ban a single IP
+
+Rack Attack supports blocking via two methods: throttling and blacklisting. In this
+case we are only concerned with blacklisting a single IP `192.168.0.1`.
+
+```
+worker$ /opt/gitlab/embedded/bin/redis-cli -h 10.0.0.1
+10.0.0.1:6379> auth <password>
+10.0.0.1:6379> setex cache:gitlab:rack::attack:allow2ban:ban:192.168.0.1 68400 "1"
+```
+
+This will ban the IP for 24 hours (68400 seconds) by storing the value "1" in the listed
+key. Since this isn't a preferred method to blacklist a host it's best not to use a longer TTL.
+
+### Should a block need to be removed
+
+Removing blocks is as simple as removing the key in Redis.
+
+```
+worker$ /opt/gitlab/embedded/bin/redis-cli -h 10.0.0.1
+10.0.0.1:6379> auth <password>
+10.0.0.1:6379> del cache:gitlab:rack::attack:allow2ban:ban:192.168.0.1
+```