Crash in sidekiq when referenced item was deleted before the job was processed
Summary
Hi, sorry if this was already reported, but this kind of issues are happening on my GitLab instance quite often. When referenced record is deleted before the queued job is processed, it crashes the sidekiq.
Steps to reproduce
Be quicker than sidekiq and delete referenced item. In the first log it was a comment and in the second one a mail - but I'm sure that this applies also to other jobs.
Expected behavior
Sidekiq should stop retrying the job.
Actual behavior
Sidekiq crashes.
Relevant logs and/or screenshots
2016-09-27T11:34:22.655Z 2652 TID-oqzn56yaw WARN: {"class":"NewNoteWorker","args":[151,{"note":"Zapoměl si tu comment :D","noteable_id":"","noteable_type":"Commit","line_code":"7054f27bc45f3bae0c528f1e02df90aaa31005e6_5_9","commit_id":"2f5baa586d76ab57f566976e4edd446a49d3ce58","type":"DiffNote","position":"{\"old_path\":\"src/main/java/cz/hryozivot/core/commands/SetSpawnCommand.java\",\"new_path\":\"src/main/java/cz/hryozivot/core/commands/SetSpawnCommand.java\",\"old_line\":5,\"new_line\":9,\"base_sha\":\"cd0b03728e4256e3b4ad0a6051c99876676f8c9d\",\"start_sha\":\"cd0b03728e4256e3b4ad0a6051c99876676f8c9d\",\"head_sha\":\"2f5baa586d76ab57f566976e4edd446a49d3ce58\"}"}],"retry":true,"queue":"default","jid":"1cd27b2b40f1a91e144c287d","created_at":1474884675.492913,"enqueued_at":1474976062.6482866,"error_message":"Couldn't find Note with 'id'=151","error_class":"ActiveRecord::RecordNotFound","failed_at":1474884688.9721205,"retry_count":14,"retried_at":1474976062.6539419}
2016-09-27T11:34:22.655Z 2652 TID-oqzn56yaw WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=151
2016-09-27T11:34:22.655Z 2652 TID-oqzn56yaw WARN: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:324:in `raise_record_not_found_exception!'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:444:in `find_one'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:423:in `find_with_ids'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:71:in `find'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:3:in `find'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:131:in `find'
/home/git/gitlab/app/workers/new_note_worker.rb:7:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:155:in `execute_job'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:136:in `block (3 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/logging.rb:32:in `with_context'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:131:in `block (2 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:171:in `stats'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:130:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:122:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:81:in `process_one'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:69:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:17:in `watchdog'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'
2016-09-26T18:38:48.345Z 2652 TID-oqzne8fq4 ActionMailer::DeliveryJob JID-c80764395228696dc9f20e7e INFO: start
2016-09-26T18:38:48.348Z 2652 TID-oqzne8fq4 ActionMailer::DeliveryJob JID-c80764395228696dc9f20e7e INFO: fail: 0.004 sec
2016-09-26T18:38:48.348Z 2652 TID-oqzne8fq4 WARN: {"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::DeliveryJob","queue":"mailers","args":[{"job_class":"ActionMailer::DeliveryJob","job_id":"fed966e5-9557-4caa-a01e-b1692a44ee9d","queue_name":"mailers","arguments":["Notify","member_access_granted_email","deliver_now","Project",101],"locale":"en"}],"retry":true,"jid":"c80764395228696dc9f20e7e","created_at":1474915128.3436985,"enqueued_at":1474915128.343998,"error_message":"Couldn't find Member with 'id'=101","error_class":"ActiveRecord::RecordNotFound","failed_at":1474915128.348413,"retry_count":0}
2016-09-26T18:38:48.348Z 2652 TID-oqzne8fq4 WARN: ActiveRecord::RecordNotFound: Couldn't find Member with 'id'=101
2016-09-26T18:38:48.348Z 2652 TID-oqzne8fq4 WARN: /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:324:in `raise_record_not_found_exception!'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:444:in `find_one'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:423:in `find_with_ids'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/finder_methods.rb:71:in `find'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:3:in `find'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:131:in `find'
/home/git/gitlab/app/mailers/emails/members.rb:73:in `member'
/home/git/gitlab/app/mailers/emails/members.rb:29:in `member_access_granted_email'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/base.rb:198:in `process_action'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/abstract_controller/base.rb:137:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionview-4.2.7.1/lib/action_view/rendering.rb:30:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/base.rb:596:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `block in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/base.rb:593:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/base.rb:584:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/message_delivery.rb:25:in `new'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/message_delivery.rb:25:in `__getobj__'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/message_delivery.rb:34:in `message'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/message_delivery.rb:85:in `deliver_now'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.7.1/lib/action_mailer/delivery_job.rb:10:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/execution.rb:32:in `block in perform_now'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:117:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:505:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:342:in `block in simple'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:497:in `block in around'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:505:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `block in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:164:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/logging.rb:43:in `block in tag_logger'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:68:in `tagged'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/logging.rb:43:in `tag_logger'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:441:in `instance_exec'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:342:in `block in simple'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:497:in `block in around'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:505:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_perform_callbacks'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/execution.rb:31:in `perform_now'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/execution.rb:21:in `execute'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activejob-4.2.7.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:155:in `execute_job'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:136:in `block (3 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/logging.rb:32:in `with_context'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:131:in `block (2 levels) in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:171:in `stats'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:130:in `block in process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:122:in `process'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:81:in `process_one'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:69:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:17:in `watchdog'
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'
Output of checks
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 3.6.0 ? ... OK (3.6.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/4 ... ok
3/5 ... ok
3/6 ... ok
5/12 ... ok
3/19 ... ok
6/21 ... ok
6/22 ... ok
2/23 ... ok
4/26 ... ok
3/27 ... ok
2/28 ... ok
1/29 ... ok
2/30 ... ok
2/31 ... ok
1/32 ... ok
5/33 ... ok
2/34 ... ok
12/35 ... ok
13/36 ... ok
13/37 ... repository is empty
3/39 ... ok
1/40 ... ok
19/43 ... repository is empty
4/44 ... ok
4/45 ... ok
4/47 ... ok
21/48 ... ok
21/49 ... ok
1/50 ... ok
21/51 ... ok
1/52 ... repository is empty
4/53 ... ok
20/54 ... ok
20/55 ... ok
4/56 ... ok
20/57 ... ok
1/58 ... repository is empty
20/59 ... ok
20/63 ... ok
20/66 ... ok
2/68 ... ok
2/69 ... ok
1/70 ... ok
1/71 ... ok
1/72 ... repository is empty
1/74 ... ok
24/76 ... ok
24/77 ... ok
1/78 ... ok
25/79 ... ok
1/80 ... ok
2/81 ... ok
1/83 ... ok
15/84 ... ok
1/85 ... ok
1/86 ... ok
26/87 ... ok
2/88 ... ok
1/89 ... repository is empty
1/90 ... ok
28/91 ... ok
28/92 ... repository is empty
28/94 ... ok
28/95 ... repository is empty
28/96 ... repository is empty
28/97 ... repository is empty
28/98 ... repository is empty
28/100 ... repository is empty
28/101 ... repository is empty
28/102 ... repository is empty
28/103 ... repository is empty
28/104 ... repository is empty
28/106 ... repository is empty
2/107 ... ok
2/108 ... ok
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/git/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ...
3/4 ... yes
3/5 ... yes
3/6 ... yes
5/12 ... yes
3/19 ... yes
6/21 ... yes
6/22 ... yes
2/23 ... yes
4/26 ... yes
3/27 ... yes
2/28 ... yes
1/29 ... yes
2/30 ... yes
2/31 ... yes
1/32 ... yes
5/33 ... yes
2/34 ... yes
12/35 ... yes
13/36 ... yes
13/37 ... yes
3/39 ... yes
1/40 ... yes
19/43 ... yes
4/44 ... yes
4/45 ... yes
4/47 ... yes
21/48 ... yes
21/49 ... yes
1/50 ... yes
21/51 ... yes
1/52 ... yes
4/53 ... yes
20/54 ... yes
20/55 ... yes
4/56 ... yes
20/57 ... yes
1/58 ... yes
20/59 ... yes
20/63 ... yes
20/66 ... yes
2/68 ... yes
2/69 ... yes
1/70 ... yes
1/71 ... yes
1/72 ... yes
1/74 ... yes
24/76 ... yes
24/77 ... yes
1/78 ... yes
25/79 ... yes
1/80 ... yes
2/81 ... yes
1/83 ... yes
15/84 ... yes
1/85 ... yes
1/86 ... yes
26/87 ... yes
2/88 ... yes
1/89 ... yes
1/90 ... yes
28/91 ... yes
28/92 ... yes
28/94 ... yes
28/95 ... yes
28/96 ... yes
28/97 ... yes
28/98 ... yes
28/100 ... yes
28/101 ... yes
28/102 ... yes
28/103 ... yes
28/104 ... yes
28/106 ... yes
2/107 ... yes
2/108 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/usr/local/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 7
Checking GitLab ... Finished
When I restart the background jobs, the sidekiq will run, but it crashes again until I delete the job from queue.
Results of GitLab environment info
System information
System: Debian unstable
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.5.1
Bundler Version:1.13.1
Rake Version: 10.5.0
Sidekiq Version:4.2.1
GitLab information
Version: 8.13.0-pre
Revision: 054f1ec
Directory: /home/git/gitlab
DB Adapter: postgresql
URL: https://gitlab.*****.cz
HTTP Clone URL: https://gitlab.*****.cz/some-group/some-project.git
SSH Clone URL: git@gitlab.*****.cz:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 3.6.0
Repository storage paths:
- default: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/local/bin/git
This is a GitLab instance installed from source and it's running for almost 2 years (I think). Also note, that this was not happening to me only on latest master version.