psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

After some issues at my machine (fsck, disc issues etc.) I have recovered and I had to upgrade gitlab without stopping the services (I was getting some weird errors). So I upgraded it with newest (7.9.2) omnibus rpm (centos 6) and now I have this on gitlab-ctl reconfigure.

Running handlers:
[2015-04-09T18:09:59+03:00] ERROR: Running exception handlers
Running handlers complete

[2015-04-09T18:09:59+03:00] ERROR: Exception handlers complete
[2015-04-09T18:09:59+03:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 3 resources updated in 9.312983322 seconds
[2015-04-09T18:09:59+03:00] ERROR: execute[create gitlab database user] (gitlab::postgresql line 160) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" ----
STDOUT: 
STDERR: psql: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
---- End output of /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" ----
Ran /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" returned 2
[2015-04-09T18:09:59+03:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[root@gitlab postgresql]# gitlab-rake gitlab:check
Checking Environment ...

Git configured for git user? ... yes

Checking Environment ... Finished

Checking GitLab Shell ...

GitLab Shell version >= 2.6.0 ? ... OK (2.6.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
Satellites access is drwxr-x---? ... yes
hooks directories in repos are links: ... rake aborted!
PG::Error: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:568:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `postgresql_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/connection_handling.rb:87:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/relation/merger.rb:121:in `merge_multi_values'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/relation/merger.rb:76:in `merge'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/relation/spawn_methods.rb:44:in `merge!'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/relation/spawn_methods.rb:33:in `merge'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:103:in `block (2 levels) in build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:102:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:102:in `inject'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:102:in `block in build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:125:in `evaluate_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/default.rb:101:in `build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/named.rb:33:in `default_scoped'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/scoping/named.rb:28:in `all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.9/lib/active_record/querying.rb:13:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:497:in `check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:364:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check
(See full trace by running task with --trace)

EDIT After adding a softlink as suggested in this answer it works but still not fully:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/users/sign_in" for 127.0.0.1 at 2015-04-13 14:25:56 +0300
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 8ms

NoMethodError (undefined method `push_events=' for #<GitlabIssueTrackerService:0x000000066f41a8>):
  app/models/project.rb:329:in `default_issue_tracker'
  app/controllers/application_controller.rb:190:in `add_gon_variables'



==> /var/log/gitlab/nginx/gitlab_access.log <==
10.154.13.245 - - [13/Apr/2015:14:25:56 +0300] "GET /users/sign_in HTTP/1.1" 500 349 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36"
10.154.13.245 - - [13/Apr/2015:14:25:56 +0300] "GET /static.css HTTP/1.1" 304 0 "https://gitlab-xxx.xxx.xxx-xxx.net/users/sign_in" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36"

What is wrong?

For me it was an issue with PostgreSql Shared Memory, this article helped me a lot http://leopard.in.ua/2013/09/05/postgresql-sessting-shared-memory. hope this can help !

This may by same issue as this one: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/552

Try to manually increase kernel shared memory limit and reconfigure / restart GitLab:

echo 17179869184 > /proc/sys/kernel/shmmax
gitlab-ctl reconfigure
gitlab-ctl restart

Pretty sure this isn't always related to the shmmax issue.

In the cases I've seen it, it's due to the gitlab-ctl reconfigure starting the postgres server, and then trying to run the "create gitlab database user" step before the postgres server has properly started. I notice it happens if the machine my gitlab vm's are running on is somewhat overloaded and seems a lot more prevalent on CentOS (which is slower anyway).

The only workaround I've found for this one is to run the gitlab-ctl reconfigure again and it should work the second time.

Still it's not a very nice user experience, could something be added into the chef recipe for starting the postgres service to not return from that step until the service is confirmed to be accepting requests? The other (potentially easier) option would be to start the postgres service earlier in the chef cookbook to give it the time it needs to start.

This is possibly fixed with @dbirch contribution in omnibus-gitlab!427 (merged) . The fix will be released with 7.14 packages.

Status changed to closed

mentioned in merge request !14785