Support connecting to Redis Sentinel
This is a WIP from: gitlab-org/gitlab-ce#3355
We have many places loading redis configuration for different components (lot's of code duplication). This is also a good opportunity to remove references to "resque" that are still present on our codebase.
The idea here is to have a single place where configuration is loaded and all components who need it will get from there.
TODO:
-
Create Gitlab::RedisConfig
and concentrate all config loading logic there -
Remove duplicated code that loads resque.yml
-
Handle Sentinel settings from our resque.yml file and fallback when needed -
Merge improvements from Gitlab::RedisConfig
toGitlab::Redis
-
Refactor to the new Gitlab::Redis
class -
Refactor mail_room.yml
moving logic to separated class -
Make sure every component can connect with sentinel when defined -
Updated install documentation -
Open a MR to Omnibus to add dependent changes there (omnibus-gitlab!882 (merged))
How to test it with GDK
Sentinel is a separate daemon that monitors any amount of redis master servers and tries to achieve consensus when a failure is detect. Slave servers are automatically discovered by sentinel.
To setup the environment follow these steps
-
Update
resque.yml
with the example fileresque.yml.example
, uncomment and adapt the sentinel part -
Duplicate
redis
folder to something likeredis-slave
-
Configure different ports for both redis (you need to listen over TCP instead of UNIX socket)
-
On the slave
redis.conf
file you must add aSLAVEOF
line pointing to the master like:slaveof 127.0.0.1 6379
-
Configure different ports in
sentinel.conf
and changemymaster
tolocalhost
and point in both to the configured redis port inredis
folder, like the example below:port 26380 sentinel monitor localhost 127.0.0.1 6379 1 sentinel down-after-milliseconds localhost 10000 sentinel config-epoch localhost 0 sentinel leader-epoch localhost 0
-
You will need to run the 2 redis process and the 2 sentinel process, so change the Procfile like the example below:
redis: redis-server /path/to/gitlab-development-kit/redis/redis.conf redis-slave: redis-server /path/to/gitlab-development-kit/redis-slave/redis.conf redis_sentinel: redis-server /path/to/gitlab-development-kit/redis/sentinel.conf --sentinel redis_sentinel_slave: redis-server /path/to/gitlab-development-kit/redis-slave/sentinel.conf --sentinel
To make sure your configuration is correct run rail console
and try:
redis = Redis.new(Gitlab::Redis.params)
redis.info
# keep this screen open and try to simulate a failover below and run again after 10-15 seconds:
redis.info
you should get in the info a different port after a few seconds delay (the failover/reconnect time).
To simulate a failover on master redis, go to the terminal and run:
# port must match your master redis port
redis-cli -h localhost -p 6379 DEBUG sleep 60
Troubleshooting
If you get an error like this one: Redis::CannotConnectError: No sentinels available.
There may be something wrong with your configuration files or can be related to this: https://github.com/redis/redis-rb/issues/531 (PR that should make things better: https://github.com/redis/redis-rb/pull/534)
It's a bit rigid the way you have to config resque.yml
and sentinel.conf
otherwhise redis-rb
will not work properly:
The hostname ('my-primary-redis') of the primary redis (in sentinel.conf
) must match the one configure in resque.yml
and it must be valid ex:
# sentinel.conf:
sentinel monitor my-primary-redis 127.0.0.1 6379 1
sentinel down-after-milliseconds my-primary-redis 10000
sentinel config-epoch my-primary-redis 0
sentinel leader-epoch my-primary-redis 0
# resque.yaml
development:
url: redis://my-primary-redis:6378
sentinels:
-
host: localhost
port: 26380 # point to sentinel, not to redis port
-
host: localhost
port: 26381 # point to sentinel, not to redis port
When in doubt, please read Redis Sentinel documentation