NotNullViolation upgrading to 9.5.0
Summary
Failed to upgrade from 8.7.3 to 9.5.0 due to NotNullViolation. We successfully upgraded to 9.4.5 instead.
Steps to reproduce
We had an existing 8.7.3 instance, running on CentOS. We tried to upgrade via yum to the latest gitlab-ce and received an error:
PG::NotNullViolation: ERROR: column "description_html" contains null values
: ALTER TABLE "appearances" ALTER "description_html" SET NOT NULL
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170809142252_cleanup_appearances_schema.rb:16:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170809142252_cleanup_appearances_schema.rb:15:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170809142252_cleanup_appearances_schema.rb:15:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
What is the current bug behavior?
The upgrade showed errors and not all migrations were applied.
What is the expected correct behavior?
The migration should work without a NotNullViolation.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
Note that we successfully upgraded to 9.4.5.
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.13.4 Sidekiq Version:5.0.0 Go Version: unknown
GitLab information Version: 9.4.5 Revision: 140292e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: HTTP Clone URL: SSH Clone URL: git@:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.3.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/1 ... ok 6/2 ... ok 26/3 ... ok 26/4 ... ok 6/5 ... ok 27/6 ... ok 6/7 ... ok 27/8 ... ok 26/10 ... ok 30/11 ... ok 30/13 ... ok 27/15 ... ok 26/16 ... ok 27/17 ... ok 27/18 ... ok 6/19 ... ok 6/20 ... ok 30/24 ... ok 6/26 ... ok 27/27 ... ok 6/30 ... ok 40/32 ... ok 49/33 ... repository is empty 35/35 ... ok 40/36 ... ok 40/37 ... ok 40/38 ... ok 40/39 ... ok 40/40 ... ok 52/41 ... ok 40/42 ... ok 40/44 ... ok 40/45 ... ok 40/46 ... ok 6/48 ... ok 59/49 ... ok 27/50 ... ok 27/51 ... ok 27/54 ... ok 27/56 ... ok 27/57 ... ok 47/58 ... ok 47/59 ... ok 30/60 ... ok 66/62 ... ok 64/64 ... ok 64/65 ... ok 7/66 ... ok 9/67 ... ok 7/68 ... ok 64/69 ... ok 68/70 ... repository is empty 9/71 ... ok 7/72 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 6/2 ... yes 26/3 ... yes 26/4 ... yes 6/5 ... yes 27/6 ... yes 6/7 ... yes 27/8 ... yes 26/10 ... yes 30/11 ... yes 30/13 ... yes 27/15 ... yes 26/16 ... yes 27/17 ... yes 27/18 ... yes 6/19 ... yes 6/20 ... yes 30/24 ... yes 6/26 ... yes 27/27 ... yes 6/30 ... yes 40/32 ... yes 49/33 ... yes 35/35 ... yes 40/36 ... yes 40/37 ... yes 40/38 ... yes 40/39 ... yes 40/40 ... yes 52/41 ... yes 40/42 ... yes 40/44 ... yes 40/45 ... yes 40/46 ... yes 6/48 ... yes 59/49 ... yes 27/50 ... yes 27/51 ... yes 27/54 ... yes 27/56 ... yes 27/57 ... yes 47/58 ... yes 47/59 ... yes 30/60 ... yes 66/62 ... yes 64/64 ... yes 64/65 ... yes 7/66 ... yes 9/67 ... yes 7/68 ... yes 64/69 ... yes 68/70 ... yes 9/71 ... yes 7/72 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.4) Active users: ... 53
Checking GitLab ... Finished
Possible fixes
In 8.7.3, we had the appearances table, but there was no description_html column. This column was added in migration 20160829114652_add_markdown_cache_columns.rb, but the column is still null. In the 20170809142252_cleanup_appearances_schema.rb migration, a not null constraint is added to the column, but this fails because the column is still null.
A default could be added to the column or it needs to be not-null before the migration is applied. In our case, only the description_html column was null, the other 4 were not null.