You have skipped 8.17.1, 8.17.2, 8.17.3, 8.17.4, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6. And at least 9.0.0 and 9.1.0 have introduced new configuration options.
The update process assumes, that you are upgrading from the latest version. If not, then you have to check each update barometer for every intermediate version.
Isn't that the point of migrations? To make the changes that are missed?
Basically we have been using gitlab for about 6 months, and every update is a mess. Maybe this policy of updating in the 22nd of each month si a bad idea, maybe if they were not so tied to a specific date they could be flexible and more reliable?
Migrations are for database changes.
Changes to the files are made during apt-get update/install. But you are responsible for changes to the configuration in /etc/gitlab/gitlab.rb.
I advice you to read the update section of the blog posts for at least the major version updates.
We are using GitLab since version 7 and did not have any update problems.
For the most of the updates you don't have to do anything else than a simple apt-get update && apt-get install gitlab-ce.
But of course, the update process assumes you are upgrading from the latest version. If not, you have to check if you have to do things manually.
Why not simply update more often (after each realase)? It takes only 3 minutes and is much less error-prone.
Yeah, lets see, I did that, back in the 8.15 and 8.16's And it broke Gitlab ability to add issues and milestones but could still browse. So at 8.17 I put gitlab on hold. Then this morning thought they might have it sorted by now, and it broke again. So it is working now with 9.1 after wasting and hour fixing it. Best to leave it alone and not fix what isn't broken.
@domnulnopcea Has a valid point, why cant the update process add to the gitlab.rb file any new items it needs with the default settings, and if it really needs a setting set to run prompt the user at update time.
Other applications are able to ask the user for info during updates
For sure, failures in update process are a worst case szenario.
But even when you running a big instance with hundrets of users and repos you should check all instructions to avoid possible errors.
Perhaps @smcgivern can give a suggestion to avoid this kind of update error?
@domnulnopcea if you have a specific problem you'd like help with, please follow the instructions at https://about.gitlab.com/support/ As you haven't actually said what the problem is, that's really all I can do to help you at the moment
@r.brown.bayliss as you've provided specific information, here's what I can say:
Where should I start looking? Ubuntu server 16.04 and omnibus edition with external ngnix.
why cant the update process add to the gitlab.rb file any new items it needs with the default settings, and if it really needs a setting set to run prompt the user at update time
gitlab.rb is a Ruby file. That means it is owned by the administrator, and we can't write to it, only read from it. You shouldn't have needed to add anything for 9.0, but please note that the 9.0 release did require downtime: https://about.gitlab.com/2017/03/22/gitlab-9-0-released/#upgrade-barometer
Fundamentally, it's as @maxraab says. You can keep up with the releases, and everything should work fine. If you don't do that, then everything should still work fine, but we do advise reading the release notes for the versions you're 'skipping' if that's the case, especially if this involves a major version upgrade
Why cant you write to it? The update process in Ubuntu is usually run with
su privileges?
Anyway, I removed gitlab, did apt-get --purge and and then reinstalled, as
I have had to do the previous 2 items I updated to get a working version.
the update s from 8.15 to 8.16 and 8.17 were not major updates yet did not
go smoothly which is why I abandoned the regular updates.
I am not even a big user, a few users and half a dozen projects we are
working on. Pretty much lost all faith and the only reason I am not
looking for alternatives is it's half past 9pm on a Tuesday.
@r.brown.bayliss we can't write to it because it's an executable Ruby file, which means that you can do anything in there as long as it's executable Ruby. But, like I said, the upgrade shouldn't require you to add anything there - but it doesn't sound like you did need to add anything there?
I'm sorry about the problems you've had, but if you don't tell us what the issues you had actually were using the troubleshooting documentation we provide, it's hard to say what the problems were caused by. I appreciate that you're not in a position to provide that now, as your installation is working now, but at least that's a positive thing
Belive me I did post in the issues at the time, and got some help that I
needed to fix it, but you guys seem to have about 10 issues forums on the
internet. Maybe make it less confusing to find in a googel search?. Cheers
No, lets reopen it, because here we are a few days latter, another
gitlab-ce update and it failed too and once again gitlab is down with a bad
gateway and this time it's not tuesday. 4 failed updates that have left it
unusable. Why?
@r.brown.bayliss as I said in the post I linked to, we need more details. I provided links on how to get those. You haven't even given the full output of the apt-get command, so it's impossible to say what's wrong - but this doesn't appear to be a common issue.
gitlab: Thank you for installing GitLab!
gitlab: To configure and start GitLab, RUN THE FOLLOWING COMMAND:
sudo gitlab-ctl reconfigure
gitlab: GitLab should be reachable at http://git..nz
gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab: And running reconfigure again.
gitlab:
gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab:
Checking PostgreSQL executables:Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab::postgresql-bin"]
Synchronizing Cookbooks:
bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if)
Recipe: gitlab::gitlab-rails
directory[/var/log/gitlab] action create (up to date)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
directory[create tmp/repositories] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
directory[/var/opt/gitlab/backups] action create (up to date)
directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
templatesymlink[Create a database.yml and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
(up to date)
templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
(up to date)
templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
(up to date)
templatesymlink[Create a aws.yml and create a symlink to Rails root] action delete
template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/aws.yml to /var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
(up to date)
templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action delete
template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
(up to date)
templatesymlink[Create a relative_url.rb and create a symlink to Rails root] action delete
template[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb to /var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
(up to date)
templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
(up to date)
templatesymlink[Create a rack_attack.rb and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb to /var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
(up to date)
templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
(up to date)
templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
(up to date)
directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
update content in file /var/opt/gitlab/postgresql/data/postgresql.conf from 4879c3 to eadb6d
--- /var/opt/gitlab/postgresql/data/postgresql.conf 2017-04-28 20:43:33.133260965 +1200
+++ /var/opt/gitlab/postgresql/data/.chef-postgresql.conf20170428-12946-awjojs 2017-04-28 20:46:06.455668085 +1200
@@ -250,7 +250,7 @@
- Planner Cost Constants -
#seq_page_cost = 1.0 # measured on an arbitrary scale
-#random_page_cost = 4.0 # same scale as above
+random_page_cost = 4.0 # same scale as above
#cpu_tuple_cost = 0.01 # same scale as above
#cpu_index_tuple_cost = 0.005 # same scale as above
#cpu_operator_cost = 0.0025 # same scale as above
@@ -377,7 +377,7 @@
#debug_print_rewritten = off
#debug_print_plan = off
#debug_pretty_print = on
-#log_checkpoints = off
+log_checkpoints = off
#log_connections = off
#log_disconnections = off
#log_duration = off
@@ -404,7 +404,7 @@
# %% = '%'
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
-#log_temp_files = -1 # log temporary files equal or larger
+log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
# -1 disables, 0 logs all temp files
#log_timezone = '(defaults to server environment setting)'
@@ -520,7 +520,7 @@
#------------------------------------------------------------------------------
#deadlock_timeout = 1s
-#max_locks_per_transaction = 64 # min 10
+max_locks_per_transaction = 64 # min 10
# (change requires restart)
Note: Each lock table slot uses ~270 bytes of shared memory, and there are
update content in file /var/opt/gitlab/gitlab-workhorse/VERSION from f1f4ee to c2ce4e
--- /var/opt/gitlab/gitlab-workhorse/VERSION 2017-04-28 20:43:43.593425189 +1200
+++ /var/opt/gitlab/gitlab-workhorse/.chef-VERSION20170428-12946-19j4y1p 2017-04-28 20:46:07.127678638 +1200
@@ -1,2 +1,2 @@
-gitlab-workhorse v1.4.3-20170422.132251
+gitlab-workhorse v1.4.3-20170426.122259
template[/var/opt/gitlab/gitlab-workhorse/config.toml] action create (up to date)
Recipe: gitlab::mailroom_disable
link[/opt/gitlab/service/mailroom] action delete (up to date)
directory[/opt/gitlab/sv/mailroom] action delete (up to date)
Recipe: gitlab::nginx_disable
link[/opt/gitlab/service/nginx] action delete (up to date)
directory[/opt/gitlab/sv/nginx] action delete (up to date)
Recipe: gitlab::remote-syslog_disable
link[/opt/gitlab/service/remote-syslog] action delete (up to date)
directory[/opt/gitlab/sv/remote-syslog] action delete (up to date)
Recipe: gitlab::logrotate
directory[/opt/gitlab/sv/logrotate] action create (up to date)
directory[/opt/gitlab/sv/logrotate/log] action create (up to date)
directory[/opt/gitlab/sv/logrotate/log/main] action create (up to date)
template[/opt/gitlab/sv/logrotate/run] action create (up to date)
template[/opt/gitlab/sv/logrotate/log/run] action create (up to date)
template[/var/log/gitlab/logrotate/config] action create (up to date)
ruby_block[reload logrotate svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/logrotate/down] action delete (up to date)
directory[/opt/gitlab/sv/logrotate/control] action create (up to date)
template[/opt/gitlab/sv/logrotate/control/t] action create (up to date)
link[/opt/gitlab/init/logrotate] action create (up to date)
link[/opt/gitlab/service/logrotate] action create (up to date)
ruby_block[supervise_logrotate_sleep] action run (skipped due to not_if)
service[logrotate] action nothing (skipped due to action :nothing)
Recipe: gitlab::mattermost_disable
link[/opt/gitlab/service/mattermost] action delete (up to date)
directory[/opt/gitlab/sv/mattermost] action delete (up to date)
Recipe: gitlab::gitlab-pages_disable
link[/opt/gitlab/service/gitlab-pages] action delete (up to date)
directory[/opt/gitlab/sv/gitlab-pages] action delete (up to date)
Recipe: gitlab::registry_disable
link[/opt/gitlab/service/registry] action delete (up to date)
directory[/opt/gitlab/sv/registry] action delete (up to date)
Recipe: gitlab::gitlab-healthcheck
template[/opt/gitlab/etc/gitlab-healthcheck-rc] action create (up to date)
Recipe: gitlab::prometheus_user
group[Prometheus user and group] action create (up to date)
user[Prometheus user and group] action create (up to date)
Recipe: gitlab::prometheus
directory[/var/opt/gitlab/prometheus] action create (up to date)
directory[/var/log/gitlab/prometheus] action create (up to date)
template[Prometheus template] action create (up to date)
directory[/opt/gitlab/sv/prometheus] action create (up to date)
directory[/opt/gitlab/sv/prometheus/log] action create (up to date)
directory[/opt/gitlab/sv/prometheus/log/main] action create (up to date)
template[/opt/gitlab/sv/prometheus/run] action create (up to date)
template[/opt/gitlab/sv/prometheus/log/run] action create (up to date)
template[/var/log/gitlab/prometheus/config] action create (up to date)
ruby_block[reload prometheus svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/prometheus/down] action delete (up to date)
link[/opt/gitlab/init/prometheus] action create (up to date)
link[/opt/gitlab/service/prometheus] action create (up to date)
ruby_block[supervise_prometheus_sleep] action run (skipped due to not_if)
service[prometheus] action nothing (skipped due to action :nothing)
Recipe: gitlab::node-exporter
directory[/var/log/gitlab/node-exporter] action create (up to date)
directory[/var/opt/gitlab/node-exporter/textfile_collector] action create (up to date)
directory[/opt/gitlab/sv/node-exporter] action create (up to date)
directory[/opt/gitlab/sv/node-exporter/log] action create (up to date)
directory[/opt/gitlab/sv/node-exporter/log/main] action create (up to date)
template[/opt/gitlab/sv/node-exporter/run] action create (up to date)
template[/opt/gitlab/sv/node-exporter/log/run] action create (up to date)
template[/var/log/gitlab/node-exporter/config] action create (up to date)
ruby_block[reload node-exporter svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/node-exporter/down] action delete (up to date)
link[/opt/gitlab/init/node-exporter] action create (up to date)
link[/opt/gitlab/service/node-exporter] action create (up to date)
ruby_block[supervise_node-exporter_sleep] action run (skipped due to not_if)
service[node-exporter] action nothing (skipped due to action :nothing)
Recipe: gitlab::redis-exporter
directory[/var/log/gitlab/redis-exporter] action create (up to date)
directory[/opt/gitlab/sv/redis-exporter] action create (up to date)
directory[/opt/gitlab/sv/redis-exporter/log] action create (up to date)
directory[/opt/gitlab/sv/redis-exporter/log/main] action create (up to date)
template[/opt/gitlab/sv/redis-exporter/run] action create (up to date)
template[/opt/gitlab/sv/redis-exporter/log/run] action create (up to date)
template[/var/log/gitlab/redis-exporter/config] action create (up to date)
ruby_block[reload redis-exporter svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/redis-exporter/down] action delete (up to date)
link[/opt/gitlab/init/redis-exporter] action create (up to date)
link[/opt/gitlab/service/redis-exporter] action create (up to date)
ruby_block[supervise_redis-exporter_sleep] action run (skipped due to not_if)
service[redis-exporter] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgres-exporter
directory[/var/log/gitlab/postgres-exporter] action create (up to date)
directory[/opt/gitlab/etc/postgres-exporter/env] action create (up to date)
file[/opt/gitlab/etc/postgres-exporter/env/DATA_SOURCE_NAME] action create (up to date)
directory[/opt/gitlab/sv/postgres-exporter] action create (up to date)
directory[/opt/gitlab/sv/postgres-exporter/log] action create (up to date)
directory[/opt/gitlab/sv/postgres-exporter/log/main] action create (up to date)
template[/opt/gitlab/sv/postgres-exporter/run] action create (up to date)
template[/opt/gitlab/sv/postgres-exporter/log/run] action create (up to date)
template[/var/log/gitlab/postgres-exporter/config] action create (up to date)
ruby_block[reload postgres-exporter svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/postgres-exporter/down] action delete (up to date)
link[/opt/gitlab/init/postgres-exporter] action create (up to date)
link[/opt/gitlab/service/postgres-exporter] action create (up to date)
ruby_block[supervise_postgres-exporter_sleep] action run (skipped due to not_if)
service[postgres-exporter] action nothing (skipped due to action :nothing)
Recipe: gitlab::gitlab-monitor
directory[/var/opt/gitlab/gitlab-monitor] action create (up to date)
directory[/var/log/gitlab/gitlab-monitor] action create (up to date)
template[/var/opt/gitlab/gitlab-monitor/gitlab-monitor.yml] action create (up to date)
directory[/opt/gitlab/sv/gitlab-monitor] action create (up to date)
directory[/opt/gitlab/sv/gitlab-monitor/log] action create (up to date)
directory[/opt/gitlab/sv/gitlab-monitor/log/main] action create (up to date)
template[/opt/gitlab/sv/gitlab-monitor/run] action create (up to date)
template[/opt/gitlab/sv/gitlab-monitor/log/run] action create (up to date)
template[/var/log/gitlab/gitlab-monitor/config] action create (up to date)
ruby_block[reload gitlab-monitor svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/gitlab-monitor/down] action delete (up to date)
link[/opt/gitlab/init/gitlab-monitor] action create (up to date)
link[/opt/gitlab/service/gitlab-monitor] action create (up to date)
ruby_block[supervise_gitlab-monitor_sleep] action run (skipped due to not_if)
service[gitlab-monitor] action nothing (skipped due to action :nothing)
Recipe: gitlab::default
link[/opt/gitlab/service/gitlab-git-http-server] action delete (up to date)
directory[/opt/gitlab/sv/gitlab-git-http-server] action delete (up to date)
Recipe: gitlab::gitlab-workhorse
service[gitlab-workhorse] action restart
restart service service[gitlab-workhorse]
Running handlers:
Running handlers complete
Chef Client finished, 10/374 resources updated in 03 seconds
gitlab Reconfigured!
Checking for an omnibus managed postgresql: OK
Checking if we already upgraded: OK
The latest version 9.6.1 is already running, nothing to do
Ensuring PostgreSQL is updated: OK
Restarting previously running GitLab services
ok: run: gitaly: (pid 13438) 1s
ok: run: gitlab-monitor: (pid 13446) 0s
ok: run: gitlab-workhorse: (pid 13421) 1s
ok: run: logrotate: (pid 13453) 1s
ok: run: node-exporter: (pid 13460) 0s
ok: run: postgres-exporter: (pid 13467) 0s
ok: run: postgresql: (pid 13353) 3s
ok: run: prometheus: (pid 13478) 1s
ok: run: redis: (pid 12929) 8s
ok: run: redis-exporter: (pid 13489) 0s
ok: run: sidekiq: (pid 13498) 1s
ok: run: unicorn: (pid 13506) 0s
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else. If you need to roll back to the previous version you can
use the database backup made during the upgrade (scroll up for the filename).
`
@r.brown.bayliss Sorry to hear that the upgrade didn't happen smoothly for you. To answer your question, gitlab-ctl reconfigure is done as part of the upgrade. As you can see in the log you provided above: "Reconfiguring GitLab to apply migrations". Everything below that line is part of the gitlab-ctl reconfigure.
PS: Please use three backticks to format logs, see https://gitlab.com/help/user/markdown for details. It increases readability of logs in issue comments very much.