almost all reque jobs fail with PostgreSQL error

Created by: ikari-pl

I got a fresh gitlab installation, created some repos, imported some more. My redis stats say:

• processed: 85
• failed: 72

almost everything fails, but probably after restarting gitlab and restarting failed tasks, some pass. On the other hand, some seem to pass multiple times (so I have the same "ikari has pushed" notification repeated 3 times, with different timestmps).

There are only two kinds of failures in the log:

• Class: PostReceive Exception: ActiveRecord::StatementInvalid Error: PG::Error: ERROR: prepared statement "a1" already exists : INSERT INTO "events" ("action", "author_id", "created_at", "data", "project_id", "target_id", "target_type", "title", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING "id"
• Class: Notify Arguments: "project_access_granted_email" Exception: ActiveRecord::StatementInvalid Error: PG::Error: ERROR: prepared statement "a1" already exists : SELECT "users_projects".* FROM "users_projects" WHERE "users_projects"."id" = $1 LIMIT 1

I have no idea what I'm doing (and what I'm doing wrong), I have no experience with ruby, unfortunately. Switching between 1, 2 and 3 workers was the only modification I tried, doesn't change much (as it seemed to work for @rbrtmlnr under similar, but solved, #1015 (closed))

Created by: riyad

I think locking (or the lack thereof) might be the issue.

By Administrator on 2012-12-06T10:35:33 (imported from GitLab project)

By Administrator on 2012-12-06T10:35:33 (imported from GitLab)

Created by: ikari-pl

I could imagine this being a transaction-related issue, but wouldn't setting worker_processes 3 to 1 in /home/gitlab/gitlab/config/unicorn.rb help if it was (only one worker doing one thing at a time)?

Edit: ah, right, I get it. One worker at a time does the first task correctly, but then the second run encounters already defined prepared statements and tries to redefine them.

By Administrator on 2012-12-06T17:46:30 (imported from GitLab project)

By Administrator on 2012-12-06T17:46:30 (imported from GitLab)

Created by: ikari-pl

After hours of digging through browser tabs, github forks and commits, trial and error code changes (for example, it helped for one failing trigger if I wrapped it in a transaction:

+    ActiveRecord::Base.transaction do
+      project.trigger_post_receive(oldrev, newrev, ref, user)
+    end

but we see this is not the way to go, since there are more of these) , google and everything, I found a simple(?) workaround(?)... Just screw 'optimization' and add prepared_statements: false to your database.yml. I don't know, maybe those are just broken in rails.

Oh, and I did a whole sequence of service stops, starts, restarts and kill -9s to get the workers to pick up the 20 queued mailer tasks.

I'm therefore not sure at all if this is a gitlab problem or whether it can be easily fixed instead of worked around.

By Administrator on 2012-12-06T20:49:57 (imported from GitLab project)

By Administrator on 2012-12-06T20:49:57 (imported from GitLab)

Created by: erroric

i think this is related to https://github.com/gitlabhq/gitlabhq/issues/1015 fix for the same bug (re-establish connection to the database after resque forks) already pulled into the master branch - https://github.com/gitlabhq/gitlabhq/pull/1655, but not yet in the stable ... For my Postgresql GitLab instance this fix solved same issue with "prepared statement"

By Administrator on 2012-12-07T19:41:55 (imported from GitLab project)

By Administrator on 2012-12-07T19:41:55 (imported from GitLab)

Created by: jozefvaclavik

+1 I'm having same issue. Strange was that I started having it only since current master branch (after 3.1 release) Until then everything was working OK.

By Administrator on 2012-12-24T11:22:23 (imported from GitLab project)

By Administrator on 2012-12-24T11:22:23 (imported from GitLab)

Created by: jozefvaclavik

Seems like disabling parepared statement solved the issue. See section 3.12.3 in Rails config doc. It should only reflect on performance.

# config/database.yml
production:
  adapter: postgresql
  prepared_statement: false

I will test it for few days and let you know. Now when I pushed commits to repo, all jobs were done successfully.

By Administrator on 2012-12-24T13:29:48 (imported from GitLab project)

By Administrator on 2012-12-24T13:29:48 (imported from GitLab)

Created by: ikari-pl

Well, for me disabling prepared statements worked, at least until the workers started to disappear from the admin panel (I can see the running processes in htop, but admin says there are no workers)

By Administrator on 2012-12-25T09:08:39 (imported from GitLab project)

By Administrator on 2012-12-25T09:08:39 (imported from GitLab)

Created by: jozefvaclavik

@ikari-pl I dont think that prepared statement should have any effect on workers visibility. Its just information for ActiveRecord postgresql adapter not to use them.

By Administrator on 2012-12-25T10:11:09 (imported from GitLab project)

By Administrator on 2012-12-25T10:11:09 (imported from GitLab)

Created by: jozefvaclavik

So disabling prepared_statement didn't help. If I restart worker, and tell him to queue all the failed jobs, then the get processed. New jobs still fail.

By Administrator on 2012-12-26T07:17:21 (imported from GitLab project)

By Administrator on 2012-12-26T07:17:21 (imported from GitLab)

Created by: jirutka

It should be solved (again) in the current master, see diff and comments under 59a86e42. If you’re running GitLab on Passenger, see also #2386.

By Administrator on 2012-12-26T17:52:20 (imported from GitLab project)

By Administrator on 2012-12-26T17:52:20 (imported from GitLab)

Created by: dzaporozhets

PostgreSQL resque fix in 4-0-stable c68540e9

By Administrator on 2012-12-26T17:59:13 (imported from GitLab project)

By Administrator on 2012-12-26T17:59:13 (imported from GitLab)

Created by: riyad

Resque has been replaced with Sidekiq.

By Administrator on 2013-01-26T20:14:14 (imported from GitLab project)

By Administrator on 2013-01-26T20:14:14 (imported from GitLab)