Skip to content
Snippets Groups Projects
New issue look: Off

error update from 9.4.5 to 9.5.1

    • Closed Issue created by username-removed-1289128 @s.bondarev

    yum install gitlab-ce and get error Checking PostgreSQL executables:/opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb:2278:in `check_version_conflict': can't activate net-ssh-4.1.0, already activated net-ssh-3.2.0 (Gem::LoadError)

    Edited by username-removed-1289128

    Designs

    Child items ...

    • Activity

    • username-removed-1289128
      username-removed-1289128 @s.bondarev ·
      Author

      Error 500 when i try to open gitlab

      production.log:

      Started HEAD "/users/sign_in" for 127.0.0.1 at 2017-08-29 02:32:10 +0300
Processing by SessionsController#new as */*
Completed 500 Internal Server Error in 101ms (ActiveRecord: 2.6ms)

ActionView::Template::Error (undefined method `password_authentication_enabled?' for #<ApplicationSetting:0x007f61e963a1f8>):
    1: %ul.nav-links.new-session-tabs.nav-tabs{ role: 'tablist' }
    2:   %li.active{ role: 'presentation' }
    3:     %a{ href: '#login-pane', data: { toggle: 'tab' }, role: 'tab' } Sign in
    4:   - if password_authentication_enabled? && signup_enabled?
    5:     %li{ role: 'presentation' }
    6:       %a{ href: '#register-pane', data: { toggle: 'tab' }, role: 'tab' } Register
  app/helpers/application_settings_helper.rb:3:in `password_authentication_enabled?'
  app/views/devise/shared/_tabs_normal.html.haml:4:in `_app_views_devise_shared__tabs_normal_html_haml__1094622066007625251_70029258976780'
  app/views/devise/sessions/new.html.haml:7:in `_app_views_devise_sessions_new_html_haml__2880652822578910150_70029258239740'
  app/controllers/sessions_controller.rb:28:in `new'
  lib/gitlab/i18n.rb:46:in `with_locale'
  lib/gitlab/i18n.rb:52:in `with_user_locale'
  app/controllers/application_controller.rb:326:in `set_locale'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
      Edited by username-removed-1289128
    • username-removed-1289128
      username-removed-1289128 @s.bondarev ·
      Author

      folow up https://gitlab.com/gitlab-org/gitlab-ce/issues/36704#note_38464154

      Edited by username-removed-1289128
    • username-removed-1289128
      username-removed-1289128 @s.bondarev ·
      Author 
      yum install gitlab-ce
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.corbina.net
 * epel: mirror.daniel-jost.net
 * extras: mirror.corbina.net
 * updates: mirror.corbina.net
Resolving Dependencies
--> Running transaction check
---> Package gitlab-ce.x86_64 0:9.4.5-ce.0.el7 will be updated
---> Package gitlab-ce.x86_64 0:9.5.1-ce.0.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================
 Package                              Arch                              Version                                      Repository                                   Size
=======================================================================================================================================================================
Updating:
 gitlab-ce                            x86_64                            9.5.1-ce.0.el7                               gitlab_gitlab-ce                            378 M

Transaction Summary
=======================================================================================================================================================================
Upgrade  1 Package

Total download size: 378 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
gitlab-ce-9.5.1-ce.0.el7.x86_64.rpm                                                                                                             | 378 MB  00:00:09
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
[SKIPPED]
Dumping pages ...
[SKIPPED]
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[DISABLED]
Creating backup archive: 1503962899_2017_08_29_9.4.5_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... skipping
  Updating   : gitlab-ce-9.5.1-ce.0.el7.x86_64                                                                                                                     1/2


       *.                  *.
      ***                 ***
     *****               *****
    .******             *******
    ********            ********
   ,,,,,,,,,***********,,,,,,,,,
  ,,,,,,,,,,,*********,,,,,,,,,,,
  .,,,,,,,,,,,*******,,,,,,,,,,,,
      ,,,,,,,,,*****,,,,,,,,,.
         ,,,,,,,****,,,,,,
            .,,,***,,,,
                ,*,.

     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


gitlab: Thank you for installing GitLab!
gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:

sudo gitlab-ctl reconfigure

gitlab: GitLab should be reachable at https://gitlab.ilp-crm.ru/
gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab: And running reconfigure again.
gitlab:
gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab:
  Cleanup    : gitlab-ce-9.4.5-ce.0.el7.x86_64                                                                                                                     2/2
Checking PostgreSQL executables:/opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb:2278:in `check_version_conflict': can't activate net-ssh-4.1.0, already activated net-ssh-3.2.0 (Gem::LoadError)
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb:1404:in `activate'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:239:in `block in finish_resolve'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:238:in `each'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:238:in `finish_resolve'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/rdoc.rb:15:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/uninstaller.rb:11:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef/provider/package/rubygems.rb:37:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef/providers.rb:79:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef.rb:25:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/bin/chef-client:23:in `<top (required)>'
        from /opt/gitlab/embedded/bin/chef-client:22:in `load'
        from /opt/gitlab/embedded/bin/chef-client:22:in `<main>'
Checking PostgreSQL executables: NOT OK
Could not update PostgreSQL executables.
Shutting down all GitLab services except those needed for migrations
ok: down: gitaly: 0s, normally up
ok: down: gitlab-monitor: 0s, normally up
ok: down: gitlab-workhorse: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: nginx: 1s, normally up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: postgresql: 0s, normally up
ok: down: prometheus: 1s, normally up
ok: down: redis: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: unicorn: 1s, normally up
ok: run: postgresql: (pid 4330) 0s
ok: run: redis: (pid 4338) 1s
run: postgresql: (pid 4330) 1s; run: log: (pid 147) 730s
run: redis: (pid 4338) 1s; run: log: (pid 131) 730s
Reconfiguring GitLab to apply migrations
/opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb:2278:in `check_version_conflict': can't activate net-ssh-4.1.0, already activated net-ssh-3.2.0 (Gem::LoadError)
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/specification.rb:1404:in `activate'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:239:in `block in finish_resolve'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:238:in `each'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems.rb:238:in `finish_resolve'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/rdoc.rb:15:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/uninstaller.rb:11:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:120:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef/provider/package/rubygems.rb:37:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef/providers.rb:79:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/lib/chef.rb:25:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/chef-12.12.15/bin/chef-client:23:in `<top (required)>'
        from /opt/gitlab/embedded/bin/chef-client:22:in `load'
        from /opt/gitlab/embedded/bin/chef-client:22:in `<main>'
Checking for an omnibus managed postgresql: OK
Checking if we already upgraded: OK
The latest version 9.6.3 is already running, nothing to do
Ensuring PostgreSQL is updated: OK
Restarting previously running GitLab services
ok: run: gitaly: (pid 4367) 1s
ok: run: gitlab-monitor: (pid 4384) 0s
ok: run: gitlab-workhorse: (pid 4387) 1s
ok: run: logrotate: (pid 4399) 0s
ok: run: nginx: (pid 4405) 1s
ok: run: node-exporter: (pid 4417) 0s
ok: run: postgres-exporter: (pid 4423) 0s
ok: run: postgresql: (pid 4330) 5s
ok: run: prometheus: (pid 4431) 1s
ok: run: redis: (pid 4338) 6s
ok: run: redis-exporter: (pid 4447) 0s
ok: run: sidekiq: (pid 4452) 1s
ok: run: unicorn: (pid 4459) 0s

Upgrade complete! If your GitLab server is misbehaving try running

   sudo gitlab-ctl restart

before anything else. If you need to roll back to the previous version you can
use the database backup made during the upgrade (scroll up for the filename).
  Verifying  : gitlab-ce-9.5.1-ce.0.el7.x86_64                                                                                                                     1/2
  Verifying  : gitlab-ce-9.4.5-ce.0.el7.x86_64                                                                                                                     2/2

Updated:
  gitlab-ce.x86_64 0:9.5.1-ce.0.el7
    • username-removed-1289128 changed the description

      changed the description

    • username-removed-1289128
      username-removed-1289128 @s.bondarev ·
      Author

      backup, remove gitlab-ce, install, restore from backup and update help me

      gitlab-rake gitlab:backup:create
mv /var/opt/gitlab/backups/* /tmp
yum remove gitlab-ce
rm -rf /var/opt/gitlab
rm -rf /opt/gitlab
yum install gitlab-ce-9.4.5
cp /tmp/1503964887_2017_08_29_9.4.5* /var/opt/gitlab/backups/
gitlab-rake gitlab:backup:restore BACKUP=1503964887_2017_08_29_9.4.5
yum install gitlab-ce
      Edited by username-removed-1289128
    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      @marin @nick.thomas I just hit this as well. It looks like omnibus-gitlab bundles net-ssh 3.2.0, but EE includes 4.1.0, so we have two versions. Can we just bump omnibus to 4.1.0?

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      Unfortunately, it looks like Chef 12.13.30 has a dependency on net-ssh < 4.0: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/Gemfile.lock#L49

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      It looks like the only reason we need chef (not chef-zero) is for chefspec, which we only need for testing. Can we just omit this and other gems from the build?

      Attempted to do this in !1903 (closed)

      Edited by Stan Hu
    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      It looks like the dependency on chef-gem is adding these extra gems into the build. Do we really need chef-gem when chef-zero is already present?

      Edited by Stan Hu
    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      Looks like chef-gem was added in back in 2015 via d1575bca:

      commit d1575bca1e34c1f866052311d24bf099381ddc45
Author: Jacob Vosmaer <contact@jacobvosmaer.nl>
Date:   Tue Sep 29 17:09:10 2015 +0200

    Use the "chef-gem" software instead of "chef"

    Hoping this will fix the ohai-chef-chef-config dependency hell.
    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      It looks like we may also be able to bump Chef to 12.21.3: https://github.com/chef/chef/commit/31b5aabd2c2ff37b0eba63033971087f1c6db000

    • Stan Hu mentioned in merge request !1905

      mentioned in merge request !1905

    • Nick Thomas
      Nick Thomas @nick.thomas ·
      Maintainer

      Good find!

      I guess omnibus's dependencies end up in the same $GEM_HOME as the gitlab-rails dependencies?

      I'd expect both ruby applications to work though, as long as both have a separate Gemfile.lock and all calls are gated through some form of bundle exec ?

    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      I guess omnibus's dependencies end up in the same $GEM_HOME as the gitlab-rails dependencies?

      @nick.thomas Correct. But I don't get why this is an issue though for the same reasons you stated. The fact that we have multiple gems in one directory shouldn't create these problems. I'll try to replicate this and investigate a bit.

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      I'd expect both ruby applications to work though, as long as both have a separate Gemfile.lock and all calls are gated through some form of bundle exec ?

      Yeah, I'm not sure if that happens in the case for gitlab-ctl (which in turn calls omnibus-ctl). https://github.com/chef/chef-dk/issues/1187 looks like a similar issue. At first, I suspected something is calling require 'net-ssh', which activates the latest version, but since the Gem spec needs 3.2.0, the version check fails. I couldn't find evidence that was happening.

      What I did find was that there are two files here:

      root@stanhu2:/opt/gitlab/embedded/lib/ruby/gems/2.3.0/specifications# ls -al | grep net-ssh
-rw-r--r-- 1 root root  2915 Aug 28 14:02 net-ssh-3.2.0.gemspec
-rw-r--r-- 1 root root  3957 Aug 28 14:02 net-ssh-4.1.0.gemspec

      If you move the net-ssh-4.1.0.gemspec file out, sudo gitlab-ctl reconfigure appears to work.

      @marin I've added your public key to ssh marin@stanhu2.gitlap.com if you want to poke around. You can just run sudo gitlab-ctl reconfigure and see the error.

      Edited by Stan Hu
    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      @stanhu Thanks, that helped a lot. I found the problem but I am now trying to wrap my head around it and how to make sense of it. The tl;dr: this is an issue in within bundler/rubygems or both.

    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      OK, in short, I think we are hitting this problem explained in bundler issue: https://github.com/bundler/bundler/issues/4602#issuecomment-228708310 . I've confirmed that if I edit chef-client binstub at /opt/gitlab/embedded/bin/chef-client and replace load Gem.activate_bin_path('chef', 'chef-client', version) with load Gem.bin_path('chef', 'chef-client', version), chef-client starts running.

      It is completely unclear to me at this moment what causes this issue or how to actually fix it. Possibly the safest bet would be to upgrade rubygems and bundler to the latest versions and start preparing a chef upgrade too.

      If that doesn't resolve it OR if we need a hotfix for customers, we can always just patch chef-client with the change.

      It is important to note that I've tried installing 9.4.5 and then upgrading to 9.5.1 but that didn't cause this issue. So this problem is related to a prior installation and upgrade, and it manifested itself now only because this specific gem is being used by chef-client

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      @marin Thanks for the investigation. That sounds like a good plan.

    • Marin Jankovski mentioned in merge request !1909 (merged)

      mentioned in merge request !1909 (merged)

    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      @stanhu Can you upgrade your instance with a package in https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/62128 ? This package contains upgraded version of rubygems, and I wonder whether this PR https://github.com/rubygems/rubygems/pull/1904/ resolves the issue.

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      @marin I think that is a CE package. I triggered an EE package build here: https://gitlab.com/gitlab-org/omnibus-gitlab/-/jobs/31357824

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      Actually, my instance is old, so I think the manual trigger only works for Ubuntu 16.04. Looking for that magic API call to trigger a EE build with that branch now...

    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      Right, sorry about that, it is a CE package. I've created a pipeline here: https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/62212

    • Stan Hu
      Stan Hu @stanhu ·
      Maintainer

      @marin I think that worked (aside from the CurrentSettings problem)!

    • Marin Jankovski
      Marin Jankovski @marin ·
      Maintainer

      Yay. The fix is merged and will be shipped with GitLab 10!

    • Marin Jankovski assigned to @marin

      assigned to @marin

    • Marin Jankovski changed milestone to %10.0

      changed milestone to %10.0

    • Marin Jankovski added Unscheduled label

      added Unscheduled label

    • Marin Jankovski closed

      closed

    Please register or sign in to reply