Redis::InheritedError (Tried to use a connection from a child process without reconnecting ...
Created by: dzaporozhets
Started GET "/admin" for 127.0.0.1 at 2012-11-20 14:50:06 +0000
Processing by Admin::DashboardController#index as HTML
Completed 500 Internal Server Error in 3ms
Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.):
app/controllers/admin/dashboard_controller.rb:3:in `index'Version: master, ed92cfde
Appeared after #1981. Reproduce: randomly
Activity
Created by: VonC
"Reproduce: randomly" ? I reproduce it every time when trying an LDAP connextion.
ece56329 was still working (and still does, I just tested it). But I suspect since 36bbcc6f and its gem 'omniauth', "
> 1.1.1" or gem "resque", "> 1.23.0", it doesn't work anymoreOther login (like admin@local.host) work. But LDAP login no longer work.
Note: I am using a Redis 2.4.17, not a Redis 2.6.4. Not sure if it is related (though I am reading http://librelist.com/browser//resque/2012/6/7/reconnect-to-redis-after-fork/).
By Administrator on 2012-11-20T16:03:28 (imported from GitLab)
Created by: dzaporozhets
@vonc I'll rollback omniauth then. I cannot test LDAP on my machine so I hope you will help me
By Administrator on 2012-11-20T16:02:25 (imported from GitLab)
Created by: VonC
Considering I should provide a test environment in two days, color me as... very motivated ;) But I don't know yet if the change comes from omniauth or resque, or another module.
The very same LDAP parameters work with ece56329, not with f6248c25 (my current master update).
By Administrator on 2012-11-20T16:06:16 (imported from GitLab)
Created by: wasilak
I have same problem, but I don't use LDAP at all. Also one time it occures and another time it doesn't. And then another refresh and it is broken... :)
Started GET "/admin" for (my IP) at 2012-11-20 23:15:37 +0100(some more logs..)
Started GET "/admin" for (my IP) at 2012-11-20 23:16:12 +0100 Processing by Admin::DashboardController#index as HTML Completed 500 Internal Server Error in 137ms Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.): app/controllers/admin/dashboard_controller.rb:3:in `index'By Administrator on 2012-11-20T22:24:07 (imported from GitLab)
Created by: VonC
Updated to latest b39002ce (Use namespace for gitlab resque) The issue persists.
Note: the last working environment used redis2.2.2, this one: 3.0.2, and its bundle/ruby/1.9.1/gems/redis-3.0.2/lib/redis.rb no longer contain
module DisableThreadSafety, which, I suspect, made that thread-safe check error invisible.Resque also changed from 1.20 (9 months ago) to 1.23 (no diff in its lib/resque.rb)
https://github.com/redis/redis-rb/issues/245#issuecomment-6257667 suggests to actually reconnect after forking(!), prompting that user to modify the initializer/resque.rb (http://stackoverflow.com/questions/10661026/deploying-redis-to-heroku-unable-to-connect)
So I need to check resque.yml or redis.yml
There should be a
Redis.newsomewhere (as in http://stackoverflow.com/questions/10922197/resque-is-not-picking-up-redis-configuration-settings), but I don't see one.By Administrator on 2012-11-21T07:28:43 (imported from GitLab)
Created by: VonC
Note: https://yourGitLabServer/admin/resque is interesting (connected as admin@local.host):
It does display the Resque error in detail:
Redis::InheritedError at /info/resque/overview Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking. file: client.rb location: ensure_connected line: 277 BACKTRACE (expand) JUMP TO: GET POST COOKIES ENV /home/auser/gitpoc/ce5/usr/local/apps/ruby-1.9.2-p320/lib/ruby/1.9.1/monitor.rb in mon_synchronize 201. yield /home/auser/gitpoc/ce5/gitlab/github/config/initializers/4_resque.rb in call 22. @app.call(env) GET No GET data. POST No POST data. COOKIES Variable Value Gitblit "dd94709528bb1c83d18f3288d4043f4742891f4f" _gitlab_session "AAh7CEkiD3Nlc3Npb25faYQGOgZFRkkiJSRjOTc5NzU5NmE0ZjRmNjhkZWY3ZmEBy Administrator on 2012-11-21T08:39:05 (imported from GitLab)
Created by: VonC
I also got the error when I log as admin (no LDAP here, usual admin@local.host login), and click immediatly on Admin.
Started GET "/admin" for 127.0.0.1 at 2012-11-21 10:20:26 +0000 Processing by Admin::DashboardController#index as HTML Completed 500 Internal Server Error in 2ms Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking.): app/controllers/admin/dashboard_controller.rb:3:in `index'I then click on "Issues" (it works), then on "Admin" again... and:
- it work
- I can logout and login as an LDAP user successfully
So there really is one step missing on the Resque workers in term of initialization (reconnect to Redis), but I don't know where.
Plus Resque1.23 is two months old, and the current code (https://github.com/defunkt/resque) has had quite a few evolution since then. Maybe integrating the latest of Reque would work, but I am not sure how to specify that in the Gemfile.
By Administrator on 2012-11-21T10:26:20 (imported from GitLab)
Created by: dzaporozhets
I pushed workaround 898d09a2 cause I am not sure i succeed with fixing it defore release
By Administrator on 2012-11-21T11:20:25 (imported from GitLab)
Created by: VonC
I have upgraded to 9477448a, and I don't see this error anymore on login. I only see it when going in the "Admin" section and selecting the "Resque Worker" for the first time.
Any subsequent selection displays the normal worker information. Both LDAP or non-LDAP logins work fine.By Administrator on 2012-11-22T08:25:46 (imported from GitLab)
Created by: riyad
I just found http://librelist.com/browser//resque/2012/6/7/reconnect-to-redis-after-fork/ which proposes a solution using
Resque.after_fork. Haven't tested it though.By Administrator on 2012-11-22T15:14:34 (imported from GitLab)
Created by: Razer6
Same behavior as mentioned by @vonc after upgrading from 3.0.3 to 3.1. The first time in entering Resque Worker in Admin area shows this error.
By Administrator on 2012-12-01T11:53:55 (imported from GitLab)
Created by: riyad
What I've found so far http://librelist.com/browser//resque/2012/6/7/reconnect-to-redis-after-fork/#f27dc0fb6823597606a1c9cce6511e47 proposes:
Resque.after_fork do Resque.redis.client.reconnect endIn resque.rake we have:
Resque.after_fork do |job| ActiveRecord::Base.establish_connection end@randx Does any of this fix the error if you put it into initializers/4_resque.rb?
By Administrator on 2012-12-06T21:11:15 (imported from GitLab)
Created by: dzaporozhets
@Riyad seems like not work
By Administrator on 2012-12-11T16:56:56 (imported from GitLab)
Created by: riyad
Does 59a86e42 fix it?
By Administrator on 2012-12-11T18:00:29 (imported from GitLab)
Created by: dzaporozhets
@Riyad I got exception only on first..second visit to admin area or resque after gitlab start
By Administrator on 2012-12-11T18:33:18 (imported from GitLab)
Created by: moritzheiber
59a86e42 indeed "fixes" the issue after encountering it once. Running 3.1 stable here.
By Administrator on 2012-12-13T11:19:55 (imported from GitLab)
Created by: ikari-pl
@Riyad Please tell me... I have applied the first fix you mentioned, I don't get the error anymore. I have worker numbers set to 2, and I see worker[0] and worker[1] in process list, but admin panel says only "0 of 1 workers busy". Is this correct?
By Administrator on 2013-01-04T21:19:46 (imported from GitLab)
