Sidekiq error on imported gitlab repo (wrong IP)

Here is the thing, on a newly installed Ubuntu14.04 server and the last gitlabCE(8.9.4).

I've created a gitlab export of a project on a previous gitlab server, and imported it on the new one. The git repo is working and gitlab seems normal. The problems are coming when trying to use the hooks, like the issue closer for example. I've checked the gitlab_issue_pattern and this is NOT the problem. (futhermore, the issue isn't occuring on an non-imported repo)

So, on a new commit with "fix #anIssueNumber" in it, it seems to start a sidekiq task to effectively close the issue, but this task is giving an error, and the issue isn't closed. I took a look at the error and it seems that the sidekiq task is trying to contact an IP that doesn't exist on my network to close the issue.

The sidekiq error: (found in sidekiq.log)

TID-1077l4 WARN: Errno::EHOSTUNREACH: No route to host - connect(2) for "192.168.1.113" port 9000

My computer IP is 192.168.1.10

My gitlab server IP is 192.168.1.90

Nothing exists on IP 192.168.1.113, and this is not the IP of the previous gitlab from which the project has been exported

I searched this IP in /etc/hosts, in gitlab.rb, in the whole postgresSQL database, in the whole /opt/gitlab folder and in the whole /var/opt/gitlab folder without success.

How can I give a good IP to the sidekiq task?

The full error from sidekiq.log (this is a call stack -> the last the higher level)

2016-07-06_09:30:31.73781 2016-07-06T09:30:31.737Z 321 TID-1077l4 ProjectWebHookWorker JID-3cbd5ecfc5f466739e18e90c INFO: start 2016-07-06_09:30:34.83354 2016-07-06T09:30:34.833Z 321 TID-1077l4 ProjectWebHookWorker JID-3cbd5ecfc5f466739e18e90c INFO: fail: 3.096 sec 2016-07-06_09:30:34.83399 2016-07-06T09:30:34.833Z 321 TID-1077l4 WARN: {"class":"ProjectWebHookWorker","args":[1,{"object_kind":"issue","user":{"name":"Administrator","username":"root","avatar_url":"http://www.gravatar.com/avatar/e64c7d89f26bd1972e8a854d13d7dd61?s=80&d=identicon"},"project":{"name":"test-project-import","description":null,"web_url":"http://my-gitlab/root/test-project-import","avatar_url":null,"git_ssh_url":"git@my-gitlab:root/test-project-import.git","git_http_url":"http://my-gitlab/root/test-project-import.git","namespace":"root","visibility_level":0,"path_with_namespace":"root/test-project-import","default_branch":"develop","homepage":"http://my-gitlab/root/test-project-import","url":"git@my-gitlab:root/test-project-import.git","ssh_url":"git@my-gitlab:root/test-project-import.git","http_url":"http://my-gitlab/root/test-project-import.git"},"object_attributes":{"id":145,"title":"Do not read, test purpose","assignee_id":null,"author_id":1,"project_id":1,"created_at":"2016-07-05 16:41:29 UTC","updated_at":"2016-07-05 16:41:29 UTC","position":0,"branch_name":null,"description":"I hope I'll be closed automatically","milestone_id":null,"state":"opened","iid":147,"updated_by_id":null,"confidential":false,"deleted_at":null,"due_date":null,"moved_to_id":null,"url":"http://my-gitlab/root/test-project-import/issues/147","action":"open"},"repository":{"name":"test-project-import","url":"git@my-gitlab:root/test-project-import.git","description":null,"homepage":"http://my-gitlab/root/test-project-import"}},"issue_hooks"],"retry":true,"queue":"project_web_hook","jid":"3cbd5ecfc5f466739e18e90c","created_at":1467736889.8525896,"enqueued_at":1467797431.7353172,"error_message":"No route to host - connect(2) for "192.168.1.113" port 9000","error_class":"Errno::EHOSTUNREACH","failed_at":1467736892.8559148,"retry_count":12,"retried_at":1467797434.832091} 2016-07-06_09:30:34.83409 2016-07-06T09:30:34.833Z 321 TID-1077l4 WARN: Errno::EHOSTUNREACH: No route to host - connect(2) for "192.168.1.113" port 9000 2016-07-06_09:30:34.83417 2016-07-06T09:30:34.834Z 321 TID-1077l4 WARN: /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:879:in initialize' 2016-07-06_09:30:34.83418 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:879:in open' 2016-07-06_09:30:34.83419 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:879:in block in connect' 2016-07-06_09:30:34.83420 /opt/gitlab/embedded/lib/ruby/2.1.0/timeout.rb:90:in block in timeout' 2016-07-06_09:30:34.83420 /opt/gitlab/embedded/lib/ruby/2.1.0/timeout.rb:100:in call' 2016-07-06_09:30:34.83421 /opt/gitlab/embedded/lib/ruby/2.1.0/timeout.rb:100:in timeout' 2016-07-06_09:30:34.83422 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:878:in connect' 2016-07-06_09:30:34.83422 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:863:in do_start' 2016-07-06_09:30:34.83423 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:852:in start' 2016-07-06_09:30:34.83423 /opt/gitlab/embedded/lib/ruby/2.1.0/net/http.rb:1375:in request' 2016-07-06_09:30:34.83424 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty/request.rb:117:in perform' 2016-07-06_09:30:34.83425 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:545:in perform_request' 2016-07-06_09:30:34.83425 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:492:in post' 2016-07-06_09:30:34.83426 /opt/gitlab/embedded/service/gitlab-rails/app/models/hooks/web_hook.rb:24:in execute' 2016-07-06_09:30:34.83427 /opt/gitlab/embedded/service/gitlab-rails/app/workers/project_web_hook_worker.rb:8:in perform' 2016-07-06_09:30:34.83428 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:152:in execute_job' 2016-07-06_09:30:34.83429 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:134:in block (2 levels) in process' 2016-07-06_09:30:34.83429 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:128:in block in invoke' 2016-07-06_09:30:34.83430 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in call' 2016-07-06_09:30:34.83430 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in block in invoke' 2016-07-06_09:30:34.83431 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/active_record.rb:6:in call' 2016-07-06_09:30:34.83432 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in block in invoke' 2016-07-06_09:30:34.83432 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in call' 2016-07-06_09:30:34.83434 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in block in invoke' 2016-07-06_09:30:34.83434 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:11:in block in call' 2016-07-06_09:30:34.83435 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/logging.rb:32:in with_context' 2016-07-06_09:30:34.83435 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/server/logging.rb:7:in call' 2016-07-06_09:30:34.83436 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:130:in block in invoke' 2016-07-06_09:30:34.83437 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:133:in call' 2016-07-06_09:30:34.83439 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/middleware/chain.rb:133:in invoke' 2016-07-06_09:30:34.83439 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:129:in block in process' 2016-07-06_09:30:34.83440 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:168:in stats' 2016-07-06_09:30:34.83441 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:128:in process' 2016-07-06_09:30:34.83442 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:80:in process_one' 2016-07-06_09:30:34.83443 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/processor.rb:68:in run' 2016-07-06_09:30:34.83443 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:17:in watchdog' 2016-07-06_09:30:34.83444 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.1.2/lib/sidekiq/util.rb:25:in `block in safe_thread'

This sounds like you have a Web hook configured for this project:

Thanks, indeed, you were right, there was a webhook targeting the IP 192.168.1.113. This hook has certainly been exported with the repo and re-imported in the new gitlab.

I removed it and Sidekiq is going a lot better (no more error).

However I was expecting my issue to be closed by the commit and it's not happening. Can this be linked to the fact that I imported the repo with a webhook in it?

So I checked it twice, and there is actually a bug.

1. create a repo MyRepoA and add a readme
2. create an issue and modify the readme with "fix #1 (closed)" in the commit message -> the issue is closed
3. export myRepoA from edit project->export
4. import it as MyRepoB, create an issue and try to close it by adding "fix #2 (closed)" to your commit message. ->the issue isn't closed

So the current thread isn't the issue, it was caused by this issue, I close it to create a proper one

Status changed to closed