Corrupt Redis install - 'Short read or OOM loading DB'
We just had a corrupted redis install due to a disk filling up and needing expanding before we could get to it.
It caused even more errors in the logs spewing out:
==> /var/log/gitlab/ci-redis/current <==
2015-01-06_08:43:33.18692 _._
2015-01-06_08:43:33.18693 _.-``__ ''-._
2015-01-06_08:43:33.18694 _.-`` `. `_. ''-._ Redis 2.8.2 (00000000/0) 64 bit
2015-01-06_08:43:33.18694 .-`` .-```. ```\/ _.,_ ''-._
2015-01-06_08:43:33.18695 ( ' , .-` | `, ) Running in stand alone mode
2015-01-06_08:43:33.18695 |`-._`-...-` __...-.``-._|'` _.-'| Port: 0
2015-01-06_08:43:33.18695 | `-._ `._ / _.-' | PID: 17484
2015-01-06_08:43:33.18696 `-._ `-._ `-./ _.-' _.-'
2015-01-06_08:43:33.18696 |`-._`-._ `-.__.-' _.-'_.-'|
2015-01-06_08:43:33.18696 | `-._`-._ _.-'_.-' | http://redis.io
2015-01-06_08:43:33.18696 `-._ `-._`-.__.-'_.-' _.-'
2015-01-06_08:43:33.18697 |`-._`-._ `-.__.-' _.-'_.-'|
2015-01-06_08:43:33.18697 | `-._`-._ _.-'_.-' |
2015-01-06_08:43:33.18697 `-._ `-._`-.__.-'_.-' _.-'
2015-01-06_08:43:33.18697 `-._ `-.__.-' _.-'
2015-01-06_08:43:33.18698 `-._ _.-'
2015-01-06_08:43:33.18698 `-.__.-'
2015-01-06_08:43:33.18699
2015-01-06_08:43:33.18699 [17484] 06 Jan 19:43:33.186 # Server started, Redis version 2.8.2
2015-01-06_08:43:33.18699 [17484] 06 Jan 19:43:33.186 # Short read or OOM loading DB. Unrecoverable error, aborting now.
...millions of times over which filled the disk again!
The only fix I could get to work was:
apt-get remove gitlab
updatedb
locate redis
# rm -rf anything to do with redis that isn't a chef recipe
apt-get install gitlab
gitlab-ctl reconfigure
gitlab-ctl start
Versions:
GitLab 7.6.3-ee (Omnibus installer) GitLab Shell 2.4.0 GitLab API v3 Ruby 2.1.4p265 Rails 4.1.1