From 97372ba70e638b6271f6a24e60f0357dd6e0b402 Mon Sep 17 00:00:00 2001
From: Pablo Carranza <pcarranza@gmail.com>
Date: Tue, 24 May 2016 18:19:57 +0100
Subject: [PATCH] Add backups runbook

---
 README.md        |  4 ++++
 howto/backups.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
 create mode 100644 howto/backups.md

diff --git a/README.md b/README.md
index 39caa57..3a96e33 100644
--- a/README.md
+++ b/README.md
@@ -134,6 +134,10 @@ The aim of this project is to have a quick guide of what to do when an emergency
 * [Reload CheckMK metrics](howto/manage-checkmk.md#reload_host_metrics)
 * [Run pgbadger to analyze queries](howto/postgresql.md#run-pgbadger-in-the-primary-database-server)
 
+### Manage backups
+
+* [Handle backups and restorations](howto/backups.md)
+
 ## General guidelines in an emergency
 
 * Confirm that it is actually an emergency, challenge this: are we losing data? Is GitLab.com not working?
diff --git a/howto/backups.md b/howto/backups.md
new file mode 100644
index 0000000..4d7827f
--- /dev/null
+++ b/howto/backups.md
@@ -0,0 +1,50 @@
+# Backups and restore
+We currenty have multiple backup solutions:
+- AWS snapshots by `ebs.gitlap.com`
+- Azure snapshots by `azure.gitlap.com`
+- GitLab backup for database and pages
+
+## AWS snapshots
+### Snapshots
+Every night on `ebs.gitlap.com` the following snapshot script `/opt/gitlab-backup/bin/gitlab-ebs-snapshot` creates snapshots for all EC2 instances.
+### Restore
+https://dev.gitlab.org/cookbooks/gitlab-backup/blob/master/doc/gitlab-ebs-snapshot.md#restoring
+
+## Azure snapshots
+### Snapshots
+Every night on `azure.gitlap.com` the following snapshot script `/opt/gitlab-backup/bin/gitlab-azure-snapshots` creates snapshots for the Azure instances mentioned in `/etc/gitlab-azure-snapshots.yml`.
+
+Just add the chef role `"role[azure-snapshot]"` to a node and snapshots will be created.
+
+### Restore
+On `azure.gitlap.com` you can use script `/opt/gitlab-backup/bin/gitlab-azure-restore` to restore the snapshots and attach the data disks to a new created instance.
+
+So to restore i.e. file-storage1.cluster.gitlab.com follow these steps:
+#### Create new node
+```
+~/chef-repo/tools/bin/azure-create-node --role backend file-storage2.cluster.gitlab.com
+```
+#### Restore snapshot and attach the data disks
+Login into `azure.gitlap.com`
+
+You can get a list of available epochs by checking the snapshot info files in /var/lib/gitlab-azure-snapshots:
+```
+ls -al /var/lib/gitlab-azure-snapshots/
+```
+Restore snapshots:
+```
+/opt/gitlab-backup/bin/gitlab-azure-restore --epoch 1463965202 --source file-storage1.cluster.gitlab.com file-storage2.cluster.gitlab.com
+```
+
+#### Activate logical volume
+Activate the logical volume in the new created node:
+```
+# LVM support
+sudo apt-get install -y lvm2
+# look for gitlab_vg on the attached drives
+sudo vgchange -ay gitlab_vg
+# make sure the mountpoint exists
+sudo mkdir -p /var/opt/gitlab
+# mount the logical volume at /var/opt/gitlab
+sudo mount /dev/gitlab_vg/gitlab_com /var/opt/gitlab
+```
-- 
GitLab