Runner can´t build

runner=9d57f83e status=500 Internal Server Error

I'm seeing this as well after upgrading from 8.8.5 to 8.9 using the docker image. This seems to be a permission problem in the registry; the logs show:

err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/p/atlas/_manifests/revisions/sha256/67751e6f5bcec21a7b4f2cfd3aaf55f3a6452c59afd2f1a067aa714a6845373c: permission denied" err.message="unknown error"

Fixing permissions via update-permissions doesn't help.

I´m not using the registry settings.

Ok thanks, I'll open a separate issue then.

My issue is now at #19053 (closed) .
You might want to include some logs from your runner, a 500 error is pretty broad.

Same thing here. Strangely its not on all builds. Only a specific project can't get build.

WARNING: Checking for builds... failed runner=7df00092 status=500 Internal Server Error

Added ~19173 label

Can you post your log/production.log?

On build start i get this error in production log:

3 cuts form production

OpenSSL::Cipher::CipherError (bad decrypt): /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in final' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in crypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:233:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:317:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:155:in block (2 levels) in attr_encrypted' /home/git/gitlab/app/models/ci/build.rb:412:in block in project_variables' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in map' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in map' /home/git/gitlab/app/models/ci/build.rb:411:in project_variables' /home/git/gitlab/app/models/ci/build.rb:122:in variables' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/delegator/plain_object.rb:6:in delegate' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:590:in delegate_attribute' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:581:in value_for' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:442:in block in serializable_hash' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in each' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in each_with_object' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:439:in serializable_hash' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-entity-0.4.8/lib/grape_entity/entity.rb:514:in to_json' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/formatter/json.rb:6:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:36:in block in after' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:35:in collect' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/formatter.rb:35:in after' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:25:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:24:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-oauth2-1.2.1/lib/rack/oauth2/server/resource.rb:20:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-oauth2-1.2.1/lib/rack/oauth2/server/resource/bearer.rb:8:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:27:in block in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:26:in catch' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/error.rb:26:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/middleware/base.rb:18:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/builder.rb:153:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/endpoint.rb:202:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/endpoint.rb:190:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:152:in block in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:96:in block in recognize' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:68:in optimized_each' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:95:in recognize' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:141:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:114:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:44:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/grape-0.13.0/lib/grape/api.rb:39:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/mapper.rb:51:in serve' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:43:in block in serve' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in each' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/journey/router.rb:30:in serve' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/routing/route_set.rb:817:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in call!' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in call' /home/git/gitlab/lib/gitlab/middleware/go.rb:16:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-cors-0.4.0/lib/rack/cors.rb:80:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-attack-4.3.1/lib/rack/attack.rb:106:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/warden-1.2.6/lib/warden/manager.rb:35:in block in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in catch' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/etag.rb:24:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/params_parser.rb:27:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/flash.rb:260:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in context' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/cookies.rb:560:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/query_cache.rb:36:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:29:in block in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:88:in run_callbacks' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in _run_call_callbacks' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:81:in run_callbacks' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/callbacks.rb:27:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/remote_ip.rb:78:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:17:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/show_exceptions.rb:30:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:38:in call_app' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:20:in block in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:68:in block in tagged' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:26:in tagged' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/tagged_logging.rb:68:in tagged' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/rack/logger.rb:20:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/request_store-1.3.0/lib/request_store/middleware.rb:9:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/request_id.rb:21:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/lock.rb:17:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sentry-raven-0.15.6/lib/raven/integrations/rack.rb:53:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:518:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:165:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in public_send' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/railtie.rb:194:in method_missing' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in block in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in each' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in call' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:580:in process_client' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in process_client' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:674:in worker_loop' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:529:in spawn_missing_workers' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:140:in start' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn_rails:209:in <top (required)>' /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/unicorn_rails:23:in load' /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/unicorn_rails:23:in '

maybe some dependencies to the "variables page" there i get following:

Completed 500 Internal Server Error in 143ms (ActiveRecord: 9.1ms)

ActionView::Template::Error (bad decrypt): 13: - if variable.id? 14: %tr 15: %td= variable.key 16: %td= variable.value 17: %td 18: = link_to namespace_project_variable_path(@project.namespace, @Project, variable), class: "btn btn-transparent btn-variable-edit" do 19: %span.sr-only app/views/projects/variables/_table.html.haml:16:in block in _app_views_projects_variables__table_html_haml__3708802409549015508_75212180' app/views/projects/variables/_table.html.haml:12:in _app_views_projects_variables__table_html_haml__3708802409549015508_75212180' app/views/projects/variables/index.html.haml:17:in _app_views_projects_variables_index_html_haml___1311328960352402588_86617380' lib/gitlab/middleware/go.rb:16:in call'

or also i seen this:

Completed 500 Internal Server Error in 161ms (ActiveRecord: 13.1ms) Started GET "/projects/myproject/builds/29856/trace.json?state=" for 127.0.0.1 at 2016-06-27 19:30:13 +0000 Processing by Projects::BuildsController#trace as JSON Parameters: {"state"=>"", "namespace_id"=>"projects", "project_id"=>"myproject", "id"=>"29856"}

This seems to have the same dependencies / reasons as #19073 (closed)

thanks @seti - I believe you are right. We'll put a fix soon.

Reassigned to @jameslopez

Status changed to closed

closing this in favour of https://gitlab.com/gitlab-org/gitlab-ce/issues/19073 - feel free to reopen if you are still experiencing this after the fix that will probably go in 8.9.3