GitLab runner hanging

I upgraded gitlab yesterday to the latest omnibus release, that went well. I then moved the setup to a new machine.

Most of GitLab is working ok, except for 1 project that does not show up. This is another issue, I see it was reported here https://gitlab.com/gitlab-org/gitlab-ce/issues/16999

My issue now is that GitLab CI is no longer working with Installed and running gitlab-ci-runners. When I start a job it shows up in the builds list, then hangs in pending state. Please see attached

TL,DR - Make sure to copy over /var/opt/gitlab/gitlab-rails/etc/secrets.yml if you don't want to reset it.

I ran the runner in debug mode using:

gitlab-ci-multi-runner --debug --log-level debug run

I see the following error:

WARNING: Checking for builds... failed runner=1c2adc60 status=500 Internal Server Error

Corresponding error in /var/log/gitlab/gitlab-rails/production.log:

OpenSSL::Cipher::CipherError (bad decrypt):
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:73:in `final'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:73:in `crypt'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:44:in `decrypt'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:197:in `decrypt'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:280:in `decrypt'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:143:in `block (2 levels) in attr_encrypted'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/ci/build.rb:415:in `block in project_variables'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in `map'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in `map'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/ci/build.rb:414:in `project_variables'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/ci/build.rb:150:in `variables'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/delegator/plain_object.rb:6:in `delegate'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:590:in `delegate_attribute'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:581:in `value_for'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:442:in `block in serializable_hash'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in `each'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in `each_with_object'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in `serializable_hash'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:514:in `to_json'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/formatter/json.rb:6:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:36:in `block in after'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:35:in `collect'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:35:in `after'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:25:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:24:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-oauth2-1.2.1/lib/rack/oauth2/server/resource.rb:20:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-oauth2-1.2.1/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:27:in `block in call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:26:in `catch'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:26:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/endpoint.rb:202:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/endpoint.rb:190:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:152:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:96:in `block in recognize'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:68:in `optimized_each'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:95:in `recognize'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:141:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:114:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:44:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:39:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/mapper.rb:51:in `serve'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in `each'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in `serve'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:817:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:16:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-cors-0.4.0/lib/rack/cors.rb:80:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-attack-4.3.1/lib/rack/attack.rb:106:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/warden-1.2.4/lib/warden/manager.rb:35:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/warden-1.2.4/lib/warden/manager.rb:34:in `catch'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/warden-1.2.4/lib/warden/manager.rb:34:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/etag.rb:24:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/flash.rb:260:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/cookies.rb:560:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/query_cache.rb:36:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:38:in `call_app'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:20:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:68:in `block in tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:26:in `tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:68:in `tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:20:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/request_store-1.3.0/lib/request_store/middleware.rb:9:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/request_id.rb:21:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/lock.rb:17:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sentry-raven-0.15.6/lib/raven/integrations/rack.rb:53:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:518:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:165:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `public_send'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `method_missing'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:580:in `process_client'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in `worker_loop'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in `spawn_missing_workers'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:540:in `maintain_worker_count'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:294:in `join'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>'
Completed 200 OK in 481ms (Views: 150.7ms | ActiveRecord: 29.8ms)

What could it be? Am I missing anything during the migration?

Thanks!

Status changed to closed

Mentioned in issue #24458 (closed)