Geo: not removing repositories from disk on secondary nodes
Summary
When a repository is removed from the primary node, the database changes are propagated but we also notify the secondary that it needs to remove the repository from disk.
The problem is that the removal worker is calling a method that doesn't exists in the FakeProject we built so it's throwing exceptions.
This part of the codebase is not covered by specs, that's why we didn't catch this first.
Steps to reproduce
Try to delete a project in primary node and check sidekiq logs on the secondary node.
Relevant logs and/or screenshots
2017-06-21T02:56:46.348Z 11783 TID-z2hss GeoRepositoryDestroyWorker JID-4a03bd4905fda467b503738a INFO: fail: 0.049 sec [22/9060]
2017-06-21T02:56:46.349Z 11783 TID-z2hss WARN: {"context":"Job raised exception","job":{"class":"GeoRepositoryDestroyWorker","args":[32,"my-new-to-be-deleted-project","brodock/my-ne
w-to-be-deleted-project"],"retry":3,"queue":"geo","jid":"4a03bd4905fda467b503738a","created_at":1498013806.2965522,"enqueued_at":1498013806.2973328,"error_message":"undefined method
`repository_storage_path' for #<GeoRepositoryDestroyWorker::FakeProject:0x0000000b1df618>","error_class":"NoMethodError","failed_at":1498013806.3484495,"retry_count":0},"jobstr":"{
\"class\":\"GeoRepositoryDestroyWorker\",\"args\":[32,\"my-new-to-be-deleted-project\",\"brodock/my-new-to-be-deleted-project\"],\"retry\":3,\"queue\":\"geo\",\"jid\":\"4a03bd4905fd
a467b503738a\",\"created_at\":1498013806.2965522,\"enqueued_at\":1498013806.2973328}"}
2017-06-21T02:56:46.349Z 11783 TID-z2hss WARN: NoMethodError: undefined method `repository_storage_path' for #<GeoRepositoryDestroyWorker::FakeProject:0x0000000b1df618>
2017-06-21T02:56:46.349Z 11783 TID-z2hss WARN: /home/git/gitlab/app/models/repository.rb:1232:in `repository_storage_path'
/home/git/gitlab/app/models/repository.rb:76:in `path_to_repo'
/home/git/gitlab/app/models/repository.rb:1195:in `refs_directory_exists?'
/home/git/gitlab/app/models/repository.rb:481:in `exists?'
/home/git/gitlab/app/models/repository.rb:58:in `block (2 levels) in cache_method'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `block in fetch'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `block in instrument'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/notifications.rb:164:in `block in instrument'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/notifications.rb:164:in `instrument'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `instrument'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:584:in `save_block_result_to_cache'
/usr/share/rvm/gems/ruby-2.3.3/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch'
/home/git/gitlab/lib/repository_cache.rb:20:in `fetch'
/home/git/gitlab/app/models/repository.rb:1153:in `cache_method_output'
/home/git/gitlab/app/models/repository.rb:58:in `block in cache_method'
/home/git/gitlab/app/models/repository.rb:371:in `before_delete'
/home/git/gitlab/app/services/projects/destroy_service.rb:137:in `flush_caches'
/home/git/gitlab/app/services/projects/destroy_service.rb:60:in `geo_replicate'
/home/git/gitlab/app/workers/geo_repository_destroy_worker.rb:10:in `perform'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:204:in `execute_job'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:175:in `block (2 levels) in process'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/home/git/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/home/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/server/active_record.rb:15:in `call'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/usr/share/rvm/gems/ruby-2.3.3/gems/sentry-raven-2.4.0/lib/raven/integrations/sidekiq.rb:7:in `call'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:174:in `block in process'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:146:in `block (6 levels) in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/job_retry.rb:97:in `local'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:145:in `block (5 levels) in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:141:in `block (4 levels) in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:220:in `stats'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:136:in `block (3 levels) in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:135:in `block (2 levels) in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/job_retry.rb:72:in `global'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:134:in `block in dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/logging.rb:32:in `with_context'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:133:in `dispatch'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:173:in `process'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:85:in `process_one'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/processor.rb:73:in `run'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:17:in `watchdog'
/usr/share/rvm/gems/ruby-2.3.3/gems/sidekiq-5.0.0/lib/sidekiq/util.rb:26:in `block in safe_thread'Possible fixes
Edited by Gabriel Mazetto