Migrating from CI to CE. Stuck on Database Encryption Key step.

Yes, in 7.12.2 secrets didn't yet exist.

You should skip that step and follow the rest of the procedure. The db_key_base will be automatically generated.

I am getting a similar error message re: config/secrets for Step II. Moving Data 2. SQL data and build traces.

tcsh 6% sudo gitlab-ci-rake backup:create
Missing `secret_key_base` or `db_key_base` for 'production' environment. The secrets will be generated and stored in `config/secrets.yml`
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - config/secrets.yml
...

Should I skip that step too?

Try to remove /opt/gitlab/embedded/service/gitlab-ci/config/initializers/secret_token.rb and retry again.

I skipped Step II 2. (same secrets error as 1.) and 3. (GitLab and CI run on the same server).

Step II 4. I moved my *_gitlab_ci_backup.tar files to /var/opt/gitlab/backups/.

Step II 5. I ran sudo gitlab-rake ci:migrate, which found nothing, as expected.

Step II 6. I ran sudo gitlab-ctl hup unicorn and sudo gitlab-ctl restart sidekiq, as instructed.

Step III I am running GitLab CI with Omnibus packages and am using the internal NGINX configuration.

After enabling rake ci:migrate in the Admin UI, I see the CI interface, but none of the build data.

Any clues as to what's up?

What is the purpose of removing /opt/gitlab/embedded/service/gitlab-ci/config/initializers/secret_token.rb?

@celiao This generates db_key_base if not found and tries to save file, this is why you see permission denied error.

@marin Maybe Omnibus should generate config/secrets.yml in 8.0?

Yes. Either generate config/secrets.yml if necessary, or suggest that people upgrade to 7.14 before 8.0.

Please see my comment about not seeing the previous CI builds after following the steps.

I removed /opt/gitlab/embedded/service/gitlab-ci/config/initializers/secret_token.rb.

Step II 1. Provided a reasonable response. As instructed, I copied gitlab_rails['db_key_base'] = nil to my /etc/gitlab/gitlab.rb file and ran sudo gitlab-ctl reconfigure. The instructions don't say to run reconfigure, but seemed logical.

Step II 2. Gave this error message:

tcsh 24% sudo gitlab-ci-rake backup:create
Applying final database migrations ...
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `initialize'
...
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:9:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Any more clues?

Should the nil have been replaced with a secret key?

CI backups with /opt/gitlab/bin/gitlab-ci-rake backup:create in cron worked fine for me before I upgraded to 8.0.2.

It appears postgresql is up and using port 5432.

tcsh 50% sudo gitlab-ctl status
run: ci-sidekiq: (pid 7279) 2s; run: log: (pid 1567) 4642069s
run: ci-unicorn: (pid 7284) 1s; run: log: (pid 1566) 4642069s
run: gitlab-git-http-server: (pid 3789) 308s; run: log: (pid 13749) 86545s
run: logrotate: (pid 3802) 307s; run: log: (pid 1562) 4642069s
run: nginx: (pid 3808) 307s; run: log: (pid 1558) 4642069s
run: postgresql: (pid 3819) 306s; run: log: (pid 1557) 4642069s
run: redis: (pid 3830) 306s; run: log: (pid 1621) 4642069s
run: sidekiq: (pid 3838) 305s; run: log: (pid 1555) 4642069s
run: unicorn: (pid 3885) 304s; run: log: (pid 1556) 4642069s
tcsh 51% sudo more /opt/gitlab/embedded/service/gitlab-ci/config/database.yml
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

production:
  adapter: postgresql
  encoding: unicode
  database: gitlab_ci_production
  pool: 10
  username: 'gitlab_ci'
  password:
  host:
  port: 5432
  socket:
  sslmode:
  sslrootcert:

/cc @marin do you have any clue why the postgres error happens?

@marin I would love to get this resolved, if you have any suggestions. postgresql is up (sudo gitlab-ctl status) and both gitlab (sudo more /opt/gitlab/embedded/service/gitlab-rails/config/database.yml) and gitlab-ci (sudo more /opt/gitlab/embedded/service/gitlab-ci/config/database.yml) use port 5432. Could it have something to do with upgrading from 7.12.2, rather than 7.14, to 8.0.2?

@celiao I am encountering the same problem. I have some old build scripts I need from the old database pre-upgrade

@maiocj Did you also upgrade from 7.12 to 8.0.2?

@celiao Yep. I am having to recreate all of my Continuous Integration tasks, and scripts, manually.

@maiocj You might be able to browse your database with a command like:

sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production

(Omnibus installation)

@celiao Do you have the same problem?

@maiocj Do you have problems with connecting to database?

@ayufan I am waiting to see if I can find a solution, so I don't have to recreate anything manually. As I mentioned above, I am stuck at Step II 2. SQL Data and build traces. I need to generate an "8.0" backup of my gitlab-ci data so I can eventually execute sudo gitlab-rake ci:migrate, but am getting errors.

tcsh 24% sudo gitlab-ci-rake backup:create
Applying final database migrations ...
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `initialize'
...
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:9:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Can you compare config/database.yml from GitLab?

cat /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
cat /opt/gitlab/embedded/service/gitlab-ci/config/database.yml
gitlab-ctl status

Sure.

tcsh 2% sudo cat /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

production:
  adapter: postgresql
  encoding: unicode
  database: gitlabhq_production
  pool: 10
  username: 'gitlab'
  password:
  host: '/var/opt/gitlab/postgresql'
  port: 5432
  socket:
  sslmode:
  sslrootcert:

tcsh 3% sudo cat /opt/gitlab/embedded/service/gitlab-ci/config/database.yml
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

production:
  adapter: postgresql
  encoding: unicode
  database: gitlab_ci_production
  pool: 10
  username: 'gitlab_ci'
  password:
  host:
  port: 5432
  socket:
  sslmode:
  sslrootcert:

tcsh 5% sudo gitlab-ctl status
run: ci-sidekiq: (pid 17482) 0s; run: log: (pid 1567) 4718300s
run: ci-unicorn: (pid 17488) 0s; run: log: (pid 1566) 4718300s
run: gitlab-git-http-server: (pid 3789) 76539s; run: log: (pid 13749) 162776s
run: logrotate: (pid 6591) 937s; run: log: (pid 1562) 4718300s
run: nginx: (pid 3808) 76538s; run: log: (pid 1558) 4718300s
run: postgresql: (pid 3819) 76537s; run: log: (pid 1557) 4718300s
run: redis: (pid 3830) 76537s; run: log: (pid 1621) 4718300s
run: sidekiq: (pid 3838) 76536s; run: log: (pid 1555) 4718300s
run: unicorn: (pid 3885) 76535s; run: log: (pid 1556) 4718300s

Can you hand fix gitlab-ci/config/database.yml adding?

  host: '/var/opt/gitlab/postgresql'

Good find. That definitely helped. However, there is another error at the end when Dumping builds ...:

tcsh 9% sudo gitlab-ci-rake backup:create
Applying final database migrations ...
== 20150630091815 AddOptionsToBuild: migrating ================================
-- add_column(:builds, :options, :text)
   -> 0.1832s
== 20150630091815 AddOptionsToBuild: migrated (0.1834s) =======================

...

-- rename_table(:variables, :ci_variables)
   -> 0.0031s
-- rename_table(:web_hooks, :ci_web_hooks)
   -> 0.0026s
== 20150914102123 MigrateCiTables: migrated (0.1144s) =========================

== 20150921081619 RenameTaggingsIdx: migrating ================================
-- remove_index(:ci_taggings, {:name=>"taggings_idx"})
   -> 0.0014s
-- add_index(:ci_taggings, [:tag_id, :taggable_id, :taggable_type, :context, :tagger_id, :tagger_type], {:unique=>true, :name=>"ci_taggings_idx"})
   -> 0.0478s
== 20150921081619 RenameTaggingsIdx: migrated (0.0493s) =======================

done
Dumping database ...
Dumping PostgreSQL database gitlab_ci_production ... [DONE]
done
Dumping builds ...
rake aborted!
Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/embedded/service/gitlab-ci/builds
/opt/gitlab/embedded/service/gitlab-ci/lib/backup/builds.rb:6:in `realpath'
/opt/gitlab/embedded/service/gitlab-ci/lib/backup/builds.rb:6:in `initialize'
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:17:in `new'
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:17:in `block (2 levels) in <top (required)>'
Tasks: TOP => backup:create
(See full trace by running task with --trace)

/opt/gitlab/embedded/service/gitlab-ci/builds doesn't exist. /opt/gitlab/embedded/service/gitlab-ci does. I could create the builds directory. Is it okay to run sudo gitlab-ci-rake backup:create again?

I am upgrading from 7.9 to 8.0.3 and I ran into the same error. (On the GitLab CE upgrade page I saw no notice that I should upgrade to 7.14 first and now it's too late to do that.)

What I did so far:

# rm /opt/gitlab/embedded/service/gitlab-ci/config/initializers/secret_token.rb
# ln -s /var/opt/gitlab/postgresql/.s.PGSQL.5432 /tmp/.s.PGSQL.5432
# gitlab-ci-rake backup:create

Now I get the same error as Celia about the missing 'builds' directory. I have these 'builds' directories under /opt/gitlab:

root@cmiclab gitlab]# find /opt/gitlab/ -name builds
/opt/gitlab/embedded/service/gitlab-rails/app/views/ci/admin/builds
/opt/gitlab/embedded/service/gitlab-rails/app/views/ci/builds
/opt/gitlab/embedded/service/gitlab-rails/builds
/opt/gitlab/embedded/service/gitlab-ci/app/views/admin/builds
/opt/gitlab/embedded/service/gitlab-ci/app/views/builds

Some have a _build.html.haml, show.html.haml and index.html.haml file. And there is '/var/opt/gitlab/gitlab-ci/builds' that is empty. Should I create a symlink to some of them or just create an empty directory as Celia suggested?

Please look at upgraded http://doc.gitlab.com/ce/migrate_ci_to_ce/README.html#troubleshooting guide.

You should ideally go through 7.14, maybe you can get back to 7.14 do gitlab-ctl reconfigure and then try to go back to 8.0?

I see, but the upgrade to gitlab-ce 8.0.3 was successful and the gitlab-ci package has been removed, I do not have it any more. Not sure how I could get back to 7.14 now. Can I just download the gitlab-ci 7.14 package and install it? Would not it conflict with gitlab-ce 8.0.3?

I ran the chown command from the toubleshooting guide, I get this:

[root@cmiclab ~]# chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds
[root@cmiclab ~]# gitlab-ci-rake backup:create
Applying final database migrations ... 
done
Dumping database ... 
Dumping PostgreSQL database gitlab_ci_production ... [DONE]
done
Dumping builds ... 
rake aborted!
Errno::ENOENT: No such file or directory @ realpath_rec - /opt/gitlab/embedded/service/gitlab-ci/builds
/opt/gitlab/embedded/service/gitlab-ci/lib/backup/builds.rb:6:in `realpath'
/opt/gitlab/embedded/service/gitlab-ci/lib/backup/builds.rb:6:in `initialize'
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:17:in `new'
/opt/gitlab/embedded/service/gitlab-ci/lib/tasks/backup.rake:17:in `block (2 levels) in <top (required)>'
Tasks: TOP => backup:create
(See full trace by running task with --trace)
[root@cmiclab ~]#

Since none of the other builds directories are symbolic links, namely:

tcsh 5% sudo find /opt/gitlab/ -name builds
/opt/gitlab/embedded/service/gitlab-ci/app/views/builds
/opt/gitlab/embedded/service/gitlab-ci/app/views/admin/builds
/opt/gitlab/embedded/service/gitlab-rails/builds
/opt/gitlab/embedded/service/gitlab-rails/app/views/ci/builds
/opt/gitlab/embedded/service/gitlab-rails/app/views/ci/admin/builds

I suspect we ought to create /opt/gitlab/embedded/service/gitlab-ci/builds, and rerun gitlab-ci-rake backup:create. Perhaps that directory was created at some point between 7.12 and 8.0.2. Do you want to try @mespak?

@ayufan

I was incorrect about the removed gitlab-ci package, as GitLab CI is integrated in the gitlab-ce package for about a year now. But I'm still not sure how I could get back to 7.14 or 7.9. I could restore the backup but then I would loose changes from today, e.g. wiki pages.

@celiao I could, I'm just afraid that the contents of some of the builds folder should be migrated and they won't.

Interestingly, the gitlab-rails/builds folder parallel to gitlab-ci/builds is empty, while the other builds/ directories aren't.

tcsh 6% sudo ls -l /opt/gitlab/embedded/service/gitlab-ci/app/views/builds
total 12
-rw-rw-r-- 1 root root 1369 Sep 25 02:10 _build.html.haml
-rw-rw-r-- 1 root root 4864 Sep 25 02:10 show.html.haml
tcsh 7% sudo ls -l /opt/gitlab/embedded/service/gitlab-ci/app/views/admin/builds
total 8
-rw-rw-r-- 1 root root 818 Sep 25 02:10 _build.html.haml
-rw-rw-r-- 1 root root 574 Sep 25 02:10 index.html.haml
tcsh 8% sudo ls -l /opt/gitlab/embedded/service/gitlab-rails/builds
total 0
tcsh 9% sudo ls -l /opt/gitlab/embedded/service/gitlab-rails/app/views/ci/builds
total 12
-rw-rw-r-- 1 root root 1347 Sep 25 02:10 _build.html.haml
-rw-rw-r-- 1 root root 4732 Sep 25 02:10 show.html.haml
tcsh 10% sudo ls -l /opt/gitlab/embedded/service/gitlab-rails/app/views/ci/admin/builds
total 8
-rw-rw-r-- 1 root root 822 Sep 25 02:10 _build.html.haml
-rw-rw-r-- 1 root root 643 Sep 25 02:10 index.html.haml

So perhaps it's okay to have /opt/gitlab/embedded/service/gitlab-ci/builds be empty. Is /opt/gitlab/embedded/service/gitlab-rails/builds empty on your system @mespak?

Can you contribute some insight, @ayufan or @marin?

Yes, it is empty.

I found this:

https://gitlab.com/axil/omnibus-gitlab/blob/892b1adb5991fe9edd6e752d559d45096adec6bd/files/gitlab-cookbooks/gitlab/recipes/gitlab-ci.rb#L186

that tells me that /opt/gitlab/embedded/service/gitlab-ci/builds should be a symlink to node['gitlab']['gitlab-ci']['builds_directory'] that is /var/opt/gitlab/gitlab-ci/builds.

So, I did this:

ln -s /var/opt/gitlab/gitlab-ci/builds /opt/gitlab/embedded/service/gitlab-ci/builds

Then the backup command succeeded.

Now I get this error when I'm trying to backup gitlab (with gitlab-rake gitlab:backup:create ...):

rake aborted!
Errno::EACCES: Permission denied @ dir_initialize - /var/opt/gitlab/gitlab-ci/builds
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/builds.rb:18:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:85:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:14:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:backup:builds:create
(See full trace by running task with --trace)

what is weird as I did what's written in the troubleshooting page:

chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds

The directory has 700 rights.

I made it 777, then the backup script succeeded and strangely it did not create any file in it.

What are the proper owner, group and permissions for that directory? Should I change it back to 700?

When you created the CI backup with gitlab-ci-rake backup:create, you mentioned no file was created (or at least remains) in gitlab-ci/builds. Was a CI backup file created anywhere?

When you created the GitLab backup with gitlab-rake gitlab:backup:create, once the permissions were corrected, was a backup created in /var/opt/gitlab/backups?

No. The gitlab-ci backup was created under /opt/gitlab/embedded/service/gitlab-ci/tmp/backups, and the gitlab-ce backup was created where I configured it in /etc/gitlab/gitlab.rb by gitlab_rails['backup_path'].

The gitlab-ci backup command printed the path at the end. The gitlab-ce command printed just the file name.

It sounds like you should be ready to go, now that the permissions are corrected and the backups are created. Have you completed the rest of the steps and can you see your pre-8.0.3 CI builds/results?

Yes, I just finished in this second. Everything looks fine. Thank you all!

I would still be interested what are the right permissions for /var/opt/gitlab/gitlab-ci/builds because 777 does not sound correct. :)

Does 775 or 755 work? Those settings match the permissions on my builds/ directories and would be safer.

I only needed to perform this additional step (aside from those in the instructions) to get our CI data migrated:

tcsh 19% sudo ln -s /var/opt/gitlab/gitlab-ci/builds /opt/gitlab/embedded/service/gitlab-ci/builds

I didn't need to change any permissions. Thanks for joining in the conversation @mespak!

I thought I might need to add something to /etc/gitlab/gitlab.rc to address setting host: '/var/opt/gitlab/postgresql' in /opt/gitlab/embedded/service/gitlab-ci/config/database.yml, but I ran sudo gitlab-ctl reconfigure and host: was still set in database.yml.

Now that directory (/var/opt/gitlab/gitlab-ci/builds) is owned by 'git', not 'gitlab-ci'. It has 700 permissions and the "gitlab-rake gitlab:backup:create SKIP=repositories,uploads" command succeeds. I did not change the owner manually, but it got changed somehow at a later step of the upgrade.

I have to think that the troubleshooting guide is wrong and the chown command should look like this:

chown -R git:git /var/opt/gitlab/gitlab-ci/builds

That is 'git' instead of 'gitlab-ci'.

@mespak First you should change permission to gitlab-ci user then change it back to git user. So yes, I need to fix that.

mentioned in issue omnibus-gitlab#844 (closed)

Status changed to closed

The most of the problems is already resolved in Omnibus and also documented in migration doc.

If new problems arise please create a new issue.

mentioned in merge request !14785

Migrating from CI to CE. Stuck on Database Encryption Key step.

Designs

Child items ...

Activity

Admin message

Admin message

Migrating from CI to CE. Stuck on Database Encryption Key step.

Activity