Failed to upgrade 7.12.2 to 7.13.2 on Centos 6.6

This looks like an issue with a missing bridge kernel module. See:

https://gitlab.com/gitlab-org/gitlab-ce/issues/2101#note_1687308

Status changed to closed

Thank you. I was finally able to start the server.

I did this, then I reinstalled and run: sudo gitlab-ctl start

Server then started up but it returned 502. After running: sudo gitlab-ctl reconfigure it finally started to work as expected.

Status changed to reopened

Glad you have it solved, I will reopen this issue to take a look if this is reproducible so I can fix it.

@marin I run into this in CentOS 6.6 on test kitchen, too. Please let me know if you need help reproducing.

Similar problem here on RHEL 6.5. Upgrade failed and then after running reconfigure a few times and restarting I thought everything worked but I'm getting 502.

sudo rpm -Uvh gitlab-ce-7.13.2-ce.0.el6.x86_64.rpm
...
Recipe: gitlab::postgresql
  * group[gitlab-psql] action create (up to date)
  * user[gitlab-psql] action create (up to date)
  * directory[/var/opt/gitlab/postgresql] action create
    - change mode from '0700' to '0755'
    - restore selinux security context
  * directory[/var/opt/gitlab/postgresql/data] action create (up to date)
  * directory[/var/log/gitlab/postgresql] action create (up to date)
  * link[/var/opt/gitlab/postgresql/data] action create (skipped due to not_if)
  * file[/var/opt/gitlab/postgresql/.profile] action create
    - change mode from '0644' to '0600'
    - restore selinux security context
  * execute[sysctl] action nothing (skipped due to action :nothing)
  * bash[add shm settings] action run (skipped due to not_if)
  * execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
  * template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create
    - update content in file /var/opt/gitlab/postgresql/data/postgresql.conf from 3a9f87 to 7533c3
    --- /var/opt/gitlab/postgresql/data/postgresql.conf 2015-08-03 04:16:41.171728975 -0400
    +++ /tmp/chef-rendered-template20150803-23788-1swz8gx       2015-08-03 05:39:10.728976565 -0400
    @@ -69,7 +69,7 @@
     # Note:  Increasing max_connections costs ~400 bytes of shared memory per
     # connection slot, plus lock space (see max_locks_per_transaction).
     #superuser_reserved_connections = 3 # (change requires restart)
    -#unix_socket_directory = ''   # (change requires restart)
    +unix_socket_directory = '/var/opt/gitlab/postgresql'   # (change requires restart)
     #unix_socket_group = ''     # (change requires restart)
     #unix_socket_permissions = 0777   # begin with 0 to use octal notation
               # (change requires restart)
    - restore selinux security context
  * service[postgresql] action restart
    - restart service service[postgresql]
  * template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
  * template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
  * directory[/opt/gitlab/sv/postgresql] action create (up to date)
  * directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
  * directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
  * template[/opt/gitlab/sv/postgresql/run] action create (up to date)
  * template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
  * template[/var/log/gitlab/postgresql/config] action create (up to date)
  * ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
  * file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
  * directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
  * template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
  * link[/opt/gitlab/init/postgresql] action create (up to date)
  * link[/opt/gitlab/service/postgresql] action create (up to date)
  * ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
  * service[postgresql] action nothing (skipped due to action :nothing)
  * execute[create gitlab database user] action run

    ================================================================================
    Error executing action `run` on resource 'execute[create gitlab database user]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" ----
    STDOUT:
    STDERR: ERROR:  role "gitlab" already exists
    ---- End output of /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" ----
    Ran /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c "CREATE USER gitlab" returned 1

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb

    170:   execute "create #{sql_user} database user" do
    171:     command "#{bin_dir}/psql --port #{pg_port} -h #{postgresql_socket_dir} -d template1 -c \"CREATE USER #{sql_user}\""
    172:     user pg_user
    173:     not_if { !pg_helper.is_running? || pg_helper.user_exists?(sql_user) }
    174:   end
    175:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb:170:in `block in from_file'

    execute("create gitlab database user") do
      action :run
      retries 0
      retry_delay 2
      default_guard_interpreter :execute
      command "/opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d template1 -c \"CREATE USER gitlab\""
      backup 5
      returns 0
      user "gitlab-psql"
      declared_type :execute
      cookbook_name "gitlab"
      recipe_name "postgresql"
      not_if { #code block }
    end

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    - execute "bash"  "/tmp/chef-script20150803-23788-zs2p1l"
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    - execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitlab::gitlab-ci
  * execute[clear the gitlab-ci cache] action run
    - execute /opt/gitlab/bin/gitlab-ci-rake cache:clear
Recipe: gitlab::database_migrations
  * bash[migrate gitlab-ci database] action run
    - execute "bash"  "/tmp/chef-script20150803-23788-xcz661"

Running handlers:
[2015-08-03T05:39:49-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-08-03T05:39:49-04:00] ERROR: Exception handlers complete
Chef Client failed. 24 resources updated in 52.848337397 seconds
[2015-08-03T05:39:50-04:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2015-08-03T05:39:50-04:00] ERROR: Found 1 errors, they are stored in the backtrace
[2015-08-03T05:39:51-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
warning: %posttrans(gitlab-ce-7.13.2-ce.0.el6.x86_64) scriptlet failed, exit status 1

sudo gitlab-ctl reconfigure
...
Recipe: gitlab::unicorn
  * directory[/var/log/gitlab/unicorn] action create (up to date)
  * directory[/opt/gitlab/var/unicorn] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/sockets] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/etc] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/unicorn.rb] action create (up to date)
  * directory[/opt/gitlab/sv/unicorn] action create (up to date)
  * directory[/opt/gitlab/sv/unicorn/log] action create (up to date)
  * directory[/opt/gitlab/sv/unicorn/log/main] action create (up to date)
  * template[/opt/gitlab/sv/unicorn/run] action create (up to date)
  * template[/opt/gitlab/sv/unicorn/log/run] action create (up to date)
  * template[/var/log/gitlab/unicorn/config] action create (up to date)
  * ruby_block[reload unicorn svlogd configuration] action nothing (skipped due to action :nothing)
  * file[/opt/gitlab/sv/unicorn/down] action delete (up to date)
  * link[/opt/gitlab/init/unicorn] action create (up to date)
  * link[/opt/gitlab/service/unicorn] action create (up to date)
  * ruby_block[supervise_unicorn_sleep] action run (skipped due to not_if)
  * service[unicorn] action nothing (skipped due to action :nothing)
  * execute[sysctl] action nothing (skipped due to action :nothing)
  * bash[add somaxconn settings] action run
    - execute "bash"  "/tmp/chef-script20150803-9530-1744x0z"
  * execute[sysctl] action run

    ================================================================================
    Error executing action `run` on resource 'execute[sysctl]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '255'
    ---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----
    STDOUT: net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    net.ipv4.tcp_rmem = 4096 262144 4194304
    net.ipv4.tcp_wmem = 4096 262144 4194304
    net.core.rmem_max = 4194304
    net.core.wmem_max = 4194304
    net.core.rmem_default = 262144
    net.core.wmem_default = 262144
    net.core.netdev_max_backlog = 30000
    net.ipv4.tcp_fin_timeout = 30
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_keepalive_intvl = 30
    net.ipv4.tcp_keepalive_probes = 5
    net.ipv4.tcp_keepalive_time = 240
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.default.secure_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 0
    net.ipv4.tcp_max_syn_backlog = 4096
    fs.file-max = 6815744
    net.ipv4.ip_local_port_range = 9000 65500
    fs.aio-max-nr = 1048576
    net.core.somaxconn = 1024
    STDERR: error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
    error: "net.bridge.bridge-nf-call-iptables" is an unknown key
    error: "net.bridge.bridge-nf-call-arptables" is an unknown key
    error: "net.ipv6.conf.default.router_solicitations" is an unknown key
    error: "net.ipv6.conf.default.accept_ra_rtr_pref" is an unknown key
    error: "net.ipv6.conf.default.accept_ra_pinfo" is an unknown key
    error: "net.ipv6.conf.default.accept_ra_defrtr" is an unknown key
    error: "net.ipv6.conf.default.autoconf" is an unknown key
    error: "net.ipv6.conf.default.dad_transmits" is an unknown key
    error: "net.ipv6.conf.default.max_addresses" is an unknown key
    ---- End output of /sbin/sysctl -p /etc/sysctl.conf ----
    Ran /sbin/sysctl -p /etc/sysctl.conf returned 255

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/unicorn.rb

     39:   execute "sysctl" do
     40:     command "/sbin/sysctl -p /etc/sysctl.conf"
     41:     action :nothing
     42:   end
     43:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/unicorn.rb:39:in `from_file'

    execute("sysctl") do
      action :nothing
      retries 0
      retry_delay 2
      default_guard_interpreter :execute
      command "/sbin/sysctl -p /etc/sysctl.conf"
      backup 5
      returns 0
      declared_type :execute
      cookbook_name "gitlab"
      recipe_name "unicorn"
    end


Running handlers:
[2015-08-03T06:13:27-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-08-03T06:13:27-04:00] ERROR: Exception handlers complete
Chef Client failed. 4 resources updated in 8.970666965 seconds
[2015-08-03T06:13:27-04:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2015-08-03T06:13:27-04:00] ERROR: Found 1 errors, they are stored in the backtrace
[2015-08-03T06:13:28-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

sudo gitlab-ctl reconfigure
...
sudo gitlab-ctl restart

Looks like this may have fixed it for me:

sudo modprobe bridge
sudo lsmod | grep bridge
sudo nano /etc/sysctl.conf
# remove somaxconn line
# comment out net.ipv6.conf lines since ipv6 disabled via /etc/modprobe.d/ipv6.conf
sudo gitlab-ctl reconfigure

I dug in to this issue a little bit. I see why it's happening but not why it's suddenly cropping up in 7.13.x and hasn't always been an issue. The erroneous sysctl configuration is in Centos 6 by default - net.bridge.... If you bring up a vanilla CentOS 6.6 box on VMware Fusion and run sysctl -p /etc/sysctl.conf you will get the error about those keys.

Prior to 7.13.x I never saw this error using VMware Fusion Centos 6.6 in Chef Test Kitchen. Could it be a change in the error handling of the reconfigure task? Maybe it ignored this error before.

FWIW The somaxconn sysctl is new in omnibus-gitlab 7.13.0.

Did omnibus-gitlab ever run sysctl -p /etc/sysctl.conf before this release?

All the way back to the first release, yes. I went looking for that and was surprised. I expected something obvious to have been added recently. I don't think the somaxconn is causing issues but it's something to look at.

mentioned in issue #662 (closed)

I can reproduce this in Centos 6 on VMware Fusion.

  * execute[sysctl] action nothing (skipped due to action :nothing)
  * bash[add somaxconn settings] action run
    - execute "bash"  "/tmp/chef-script20150814-1568-1iaaexr"
  * execute[sysctl] action run
    
    ================================================================================
    Error executing action `run` on resource 'execute[sysctl]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '255'
    ---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----
    STDOUT: net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    net.core.somaxconn = 1024
    STDERR: error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
    error: "net.bridge.bridge-nf-call-iptables" is an unknown key
    error: "net.bridge.bridge-nf-call-arptables" is an unknown key
    ---- End output of /sbin/sysctl -p /etc/sysctl.conf ----
    Ran /sbin/sysctl -p /etc/sysctl.conf returned 255
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/unicorn.rb
    
     39:   execute "sysctl" do
     40:     command "/sbin/sysctl -p /etc/sysctl.conf"
     41:     action :nothing
     42:   end
     43:

mentioned in merge request !470 (merged)

Status changed to closed by commit 54810245

Failed to upgrade 7.12.2 to 7.13.2 on Centos 6.6

`sudo yum install gitlab-ce`

`sudo gitlab-ctl reconfigure`

Designs

Child items 0

Activity

Admin message

Admin message

Failed to upgrade 7.12.2 to 7.13.2 on Centos 6.6

sudo yum install gitlab-ce

sudo gitlab-ctl reconfigure

Activity

`sudo yum install gitlab-ce`

`sudo gitlab-ctl reconfigure`