Sidekiq error when trying to expire cache
Zendesk: https://gitlab.zendesk.com/agent/tickets/19704
Customer noticed some of these backtraces in Sidekiq logs during recent issues. It appears that we're unable to expire some cache in Redis.
cc/ @yorickpeterse @rspeicher Any ideas?
2016-04-09_20:38:47.65177 2016-04-09T20:38:47.651Z 20041 TID-33ff4 WARN: Thread TID-t09ck
2016-04-09_20:38:47.65186 2016-04-09T20:38:47.651Z 20041 TID-33ff4 WARN: /opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:185:in `lock'
2016-04-09_20:38:47.65186 /opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:185:in `mon_enter'
2016-04-09_20:38:47.65187 /opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:209:in `mon_synchronize'
2016-04-09_20:38:47.65187 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
2016-04-09_20:38:47.65188 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-3.2.2/lib/redis.rb:453:in `del'
2016-04-09_20:38:47.65189 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:41:in `del'
2016-04-09_20:38:47.65189 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:245:in `block in d
elete_entry'
2016-04-09_20:38:47.65189 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `call'
2016-04-09_20:38:47.65190 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
2016-04-09_20:38:47.65190 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:245:in `delete_ent
ry'
2016-04-09_20:38:47.65191 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:402:in `block in delete'
2016-04-09_20:38:47.65191 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `block in instrument'
2016-04-09_20:38:47.65192 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:166:in `instrument'
2016-04-09_20:38:47.65192 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:547:in `instrument'
2016-04-09_20:38:47.65193 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.2/lib/active_support/cache.rb:401:in `delete'
2016-04-09_20:38:47.65193 /opt/gitlab/embedded/service/gitlab-rails/lib/repository_cache.rb:15:in `expire'
2016-04-09_20:38:47.65193 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:302:in `block in expire_branch_cache'
2016-04-09_20:38:47.65194 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:301:in `each'
2016-04-09_20:38:47.65194 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:301:in `expire_branch_cache'
2016-04-09_20:38:47.65195 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:288:in `expire_cache'
2016-04-09_20:38:47.65195 /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:416:in `after_push_commit'
2016-04-09_20:38:47.65195 /opt/gitlab/embedded/service/gitlab-rails/app/services/git_push_service.rb:20:in `execute'
2016-04-09_20:38:47.65195 /opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:50:in `block in process_project_changes'
2016-04-09_20:38:47.65195 /opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:37:in `each'
2016-04-09_20:38:47.65196 /opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:37:in `process_project_changes'
2016-04-09_20:38:47.65196 /opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:29:in `perform'
2016-04-09_20:38:47.65196 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-04-09_20:38:47.65196 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-04-09_20:38:47.65197 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-04-09_20:38:47.65197 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-04-09_20:38:47.65197 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-04-09_20:38:47.65197 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call'
2016-04-09_20:38:47.65198 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-04-09_20:38:47.65198 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-04-09_20:38:47.65198 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-04-09_20:38:47.65199 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-04-09_20:38:47.65199 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-04-09_20:38:47.65199 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-04-09_20:38:47.65199 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-04-09_20:38:47.65200 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-04-09_20:38:47.65200 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-04-09_20:38:47.65200 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-04-09_20:38:47.65200 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-04-09_20:38:47.65201 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-04-09_20:38:47.65201 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-04-09_20:38:47.65201 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-04-09_20:38:47.65201 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-04-09_20:38:47.65201 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-04-09_20:38:47.65202 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-04-09_20:38:47.65202 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'