Skip to content
Snippets Groups Projects

Gitlab in Docker. fail to start postgresql. Error: could not create shared memory segment: Invalid argument

Summary

Want to run gitlab in docker container. Use official image gitlab/gitlab-ce:latest and docker-compose Service fail to start. Errors in PostgreSQL logs

Ubuntu 14.04 LTS
Gitlab 8.13.1 (gitlab/gitlab-ce:latest)
Docker 1.12.3
Docker-Compose 1.8.1

Steps to reproduce

create docker-compose.yml like this cat docker-compose.yml

developer@gitlab:~$ 
web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
# trying to fix error  :(
#  mem_limit: 5GB
#  shm_size: 2GB
#  ipc: host
  hostname: 'gitlab'
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      external_url 'http://gitlab'
      gitlab_rails['omniauth_enabled'] = true
      gitlab_rails['omniauth_allow_single_sign_on'] = false
      gitlab_rails['block_auto_created_users'] = true
      postgresql['shared_buffers'] = "2048MB" # recommend value is 1/4 of total RAM, up to 14GB.
      # Add any other gitlab.rb configuration here, each on its own line
  ports:
    - '8888:80'
    - '4444:443'
    - '122:22'
  volumes:
    - '/home/developer/gitlab-data-files/config:/etc/gitlab'
    - '/home/developer/gitlab-data-files/logs:/var/log/gitlab'
    - '/home/developer/gitlab-data-files/data:/var/opt/gitlab'

run service docker-compose up -d

Expected behavior

Gitlab working =-) curl localhost:8888

<html><body>You are being <a href="http://localhost/users/sign_in">redirected</a>.</body></html>

Actual behavior

Gitlab does not working =-( curl localhost:8888

<!DOCTYPE html>
<html>
<head>
  <meta content="width=device-width, initial-scale=1, maximum-scale=1" name="viewport">
  <title>Something went wrong (500)</title>
...

Relevant logs and/or screenshots

docker exec -ti developer_web_1 /bin/bash commands below executed in container

Gitlab status gitlab-ctl status

run: gitlab-workhorse: (pid 379) 1018s; run: log: (pid 378) 1018s
run: logrotate: (pid 370) 1018s; run: log: (pid 368) 1018s
run: nginx: (pid 374) 1018s; run: log: (pid 373) 1018s
down: postgresql: 1s, normally up, want up; run: log: (pid 375) 1018s
run: redis: (pid 366) 1018s; run: log: (pid 365) 1018s
run: sidekiq: (pid 2666) 3s; run: log: (pid 363) 1018s
run: unicorn: (pid 369) 1018s; run: log: (pid 367) 1018s

PG logs tail /var/log/gitlab/postgresql/current

2016-10-31_08:32:10.70156 FATAL:  could not create shared memory segment: Invalid argument
2016-10-31_08:32:10.70159 DETAIL:  Failed system call was shmget(key=5432001, size=1525014528, 03600).
2016-10-31_08:32:10.70159 HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMMAX.  To reduce the request size (currently 1525014528 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2016-10-31_08:32:10.70160       If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the
request size or reconfiguring SHMMIN is called for.
2016-10-31_08:32:10.70161       The PostgreSQL documentation contains more information about shared memory configuration.

shmmax seems ok cat /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf

kernel.shmmax = 17179869184

Config seems ok cat /etc/gitlab/gitlab.rb | grep -Ev "^$|^#"

external_url 'http://gitlab'
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = false
gitlab_rails['block_auto_created_users'] = true
postgresql['shared_buffers'] = "2048MB" # recommend value is 1/4 of total RAM, up to 14GB.

check version docker version

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

check version docker-compose --version

docker-compose version 1.8.1, build 878cff1

Output of checks

Results of GitLab application Check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

oot@gitlab:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...

GitLab Shell version >= 3.6.6 ? ... OK (3.6.6)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:242:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `postgresql_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:87:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:93:in `columns'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/acts-as-taggable-on-4.0.0/lib/acts_as_taggable_on/taggable/cache.rb:37:in `columns'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attributes.rb:98:in `columns_hash'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:969:in `block in create_binds'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `partition'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:967:in `create_binds'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:954:in `build_where'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:584:in `where!'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/query_methods.rb:574:in `where'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:10:in `where'
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:192:in `block in <class:Project>'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (3 levels) in build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:302:in `scoping'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:105:in `block (2 levels) in build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `inject'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:104:in `block in build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:127:in `evaluate_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/default.rb:103:in `build_default_scope'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:33:in `default_scoped'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/scoping/named.rb:28:in `all'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:13:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:469:in `check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:348:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:check => gitlab:gitlab_shell:check
(See full trace by running task with --trace)

(we will only investigate if the tests are passing)

Results of GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:env:info)

root@gitlab:/# gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:1.13.5
Rake Version:   10.5.0
Sidekiq Version:4.2.1
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:242:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `postgresql_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_handling.rb:87:in `connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:31:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)

Possible fixes

Look like same issues gitlab-org/gitlab-ce#1834 gitlab-org/gitlab-ce#1817 gitlab-org/gitlab-ce#522

If i set postgresql['shared_buffers'] = "10MB" - service start normally, but recommended value is much bigger. From Documentation for PG 9.2

If you have a dedicated database server with 1GB or more of RAM, a reasonable starting value for shared_buffers is 25% of the memory in your system.

Also i install gitlab omnibus on same host directly and it start normally with postgresql['shared_buffers'] = "2048MB", but it is preferable to use docker images.