Gitlab reconfigure fails on the database server because it tries to read the application configuration from the redis cache and it fails
This is a chicken and egg problem, to read the cached configuration it needs to reach redis.
This cache layer should fail softly and not prevent the migrations to be load.
We walked around this issue by providing a redis configuration to the db hosts, which doesn't make sense at all.
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
Errno::ENOENT: No such file or directory - connect(2) for /var/opt/gitlab/redis/redis.socket
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/connection/ruby.rb:205:in `connect_nonblock'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/connection/ruby.rb:205:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/connection/ruby.rb:256:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:336:in `establish_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:101:in `block in connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:293:in `with_reconnect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:100:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:364:in `ensure_connected'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:221:in `block in process'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:306:in `logging'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:220:in `process'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis/client.rb:120:in `call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis.rb:862:in `block in get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis.rb:58:in `block in synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis.rb:58:in `synchronize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.3.0/lib/redis.rb:861:in `get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/interface.rb:5:in `get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:17:in `get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:21:in `block in get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:21:in `get'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:230:in `block in read_entry'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `call'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:230:in `read_entry'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:558:in `block in find_cached_entry'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `block in instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:556:in `find_cached_entry'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:293:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:92:in `current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:10:in `current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:11:in `settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:23:in `enabled?'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:129:in `<module:Metrics>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:2:in `<module:Gitlab>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:360:in `require_or_load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:494:in `load_missing_constant'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:184:in `const_missing'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/metrics.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `block in load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:652:in `block in load_config_initializer'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:651:in `load_config_initializer'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `block in <class:Engine>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `tsort_each_child'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:328:in `require_environment!'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)