Error importing GitLab Project: bad decrypt
Summary
This issue might be relatated to the following:
I've launched a second, new GitLab instace and wanted to export some projects from my old to the new one. One project import fails with bad decrypt.
I'm running both GitLab's in docker containers behind a reverse proxy.
Both instances are running at this point the latest GitLab version: 8.13.5.
Steps to reproduce
Export a project from the latest GitLab version which has a size of about ~10MB. Try to import it into another GitLab instance.
Expected behavior
They import may take some time, due to the size of the project, but finishes successful.
Actual behavior
The import fails with this message:
Clicking on Start import will lead to an 500 error.
Relevant logs and/or screenshots
==> /var/log/gitlab/gitlab-rails/production.log <==
gitlab | Import/Export error raised on /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:26:in `rescue in restore': bad decrypt
gitlab | ["/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:233:in `decrypt'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:317:in `decrypt'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:155:in `block (2 levels) in attr_encrypted'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:75:in `block in attr_encrypted'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:54:in `public_send'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:54:in `_assign_attribute'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:41:in `block in assign_attributes'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `each'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `assign_attributes'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:29:in `perform_attribute_assignment'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:566:in `init_attributes'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:281:in `initialize'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/relation_factory.rb:194:in `existing_or_new_object'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/relation_factory.rb:156:in `imported_object'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/relation_factory.rb:107:in `generate_imported_object'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/relation_factory.rb:45:in `create'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/relation_factory.rb:28:in `create'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:118:in `block in create_relation'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:117:in `map'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:117:in `create_relation'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:55:in `block in create_relations'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:49:in `each'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:49:in `create_relations'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:23:in `block in restore'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/no_touching.rb:29:in `apply_to'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/no_touching.rb:22:in `no_touching'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:22:in `restore'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/importer.rb:12:in `each'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/importer.rb:12:in `all?'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/importer.rb:12:in `execute'", "/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:61:in `import_data'", "/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:19:in `execute'", "/opt/gitlab/embedded/service/gitlab-rails/app/workers/repository_import_worker.rb:18:in `perform'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:155:in `execute_job'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:136:in `block (3 levels) in process'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'", "/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/logging.rb:32:in `with_context'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:131:in `block (2 levels) in process'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:171:in `stats'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:130:in `block in process'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq.rb:35:in `block in <module:Sidekiq>'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:122:in `process'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:81:in `process_one'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/processor.rb:69:in `run'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:17:in `watchdog'", "/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq/util.rb:25:in `block in safe_thread'"]
gitlab |
gitlab | ==> /var/log/gitlab/sidekiq/current <==
gitlab | 2016-11-13_14:41:36.19651 2016-11-13T14:41:36.196Z 726 TID-ouixyrp1s RepositoryImportWorker JID-5b186e5f6d525a10a3ef2f03 INFO: done: 27.956 sec
gitlab |
gitlab | ==> /var/log/gitlab/sshd/current <==
gitlab | 2016-11-13_14:41:37.28053 Invalid user admin from <my.public.ip>
gitlab | 2016-11-13_14:41:37.28418 input_userauth_request: invalid user admin [preauth]
gitlab | 2016-11-13_14:41:37.32594 Received disconnect from <my.public.ip> port 38793:11: Bye Bye [preauth]
gitlab | 2016-11-13_14:41:37.32600 Disconnected from <my.public.ip> port 38793 [preauth]I've also tried what @markglenfletcher proposed with the following result:
p = Project.find_with_namespace('jdoubleu/projectname')
=> #<Project id: 31, name: "projectname", path: "projectname", description: nil, created_at: "2016-11-13 14:54:11", updated_at: "2016-11-13 14:54:42", creator_id: 2, namespace_id: 2, last_activity_at: "2016-11-13 14:54:11", import_url: nil, visibility_level: 0, archived: false, avatar: nil, import_status: "failed", repository_size: 0.0, star_count: 0, import_type: "gitlab_project", import_source: "/var/opt/gitlab/gitlab-rails/shared/tmp/project_ex...", commit_count: 0, import_error: "bad decrypt", ci_id: nil, shared_runners_enabled: true, runners_token: "AnDJbemfSyg5s-6to6RK", build_coverage_regex: nil, build_allow_git_fetch: true, build_timeout: 3600, pending_delete: false, public_builds: true, last_repository_check_failed: nil, last_repository_check_at: nil, container_registry_enabled: true, only_allow_merge_if_build_succeeds: false, has_external_issue_tracker: false, repository_storage: "default", request_access_enabled: true, has_external_wiki: false, lfs_enabled: nil, description_html: nil>
irb(main):009:0> p.import_data
=> nilResults of GitLab application Check
# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 3.6.6 ? ... OK (3.6.6)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... no
User id for git: 998. Groupd id for git: 998
Try fixing it:
sudo chown -R git:git /var/opt/gitlab/git-data/repositories
For more information see:
doc/install/installation.md in section "GitLab Shell"
Please fix the error above and rerun the checks.
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
9/2 ... ok
9/3 ... ok
7/4 ... ok
9/5 ... ok
9/6 ... ok
8/7 ... ok
8/8 ... ok
8/9 ... ok
8/10 ... ok
8/12 ... ok
9/13 ... ok
8/14 ... ok
8/15 ... ok
8/16 ... ok
8/17 ... ok
7/18 ... ok
8/19 ... ok
9/20 ... ok
8/21 ... ok
8/22 ... ok
9/24 ... ok
9/25 ... ok
2/31 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
2/1 ... yes
9/2 ... yes
9/3 ... yes
7/4 ... yes
9/5 ... yes
9/6 ... yes
8/7 ... yes
8/8 ... yes
8/9 ... yes
8/10 ... yes
8/12 ... yes
9/13 ... yes
8/14 ... yes
8/15 ... yes
8/16 ... yes
8/17 ... yes
7/18 ... yes
8/19 ... yes
9/20 ... yes
8/21 ... yes
8/22 ... yes
9/24 ... yes
9/25 ... yes
2/31 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 4
Checking GitLab ... FinishedTemporary fix
The import will create issues so I created a bare repository in this project and pushed everything via git.
Activity
Added ~12980 regression labels
Added import label
Hi @jdoubleu, did you backup your
/etc/gitlab/gitlab-secrets.jsonfile on your old instance and restore on the new one?-
@jdoubleu Right. I think that the project that you are trying to import contains secret variables, which are encrypted when stored in the database. Upon attempting to import the project these values can't be decrypted due to the differing keys.
Would you be able to try removing the secret variables from the source project and see whether the import then succeeds? Thanks!
Milestone changed to %8.14
@markglenfletcher I also ran into this error just moving a project between servers. My project turned out to have a secret variable, removed it from the source project and re-exported/imported and it imported successfully. I'm not moving between servers so I don't need to copy the gitlab-secrets around, but that seems like a pretty bad way to deal with secret variables.
-
@sidewinder12s Thanks for confirming
gitlab-secretsfile should also be migrated.@jameslopez How should we deal with this situation?
@markglenfletcher I can also confirm this. I forked my source project, so all variables were deleted in the fork, then I exported and imported it successfully.
First of all I think the export info should be extended to point to the variables:
But this would only make sense if the variables shouldn't be exported.The second thing is, that the error message which appears while or after the importing failed should be clearer.
How should we deal with this situation?
This is an idea and suggestion: According to the user permissions the only roles who can export a projects are the Master and the Owner. So no third party will ever have access to the variables. It would be a nice feature to ex- and import those variables too. Maybe you can determine a "transfer password". When the owner (or master) tries to export a project which has secret variables, those will be decrypted and, only for the transport, encrypted with a password provided by the user (or generated randomly). The import checks if it has a password, or anything else, and prompts the user for it.
This may be done in low level, so only some database entries in the database dump will be encrypted temporary or the whole package can be password protected which would be nice for others things being improted, too.
Edited by username-removed-301332Mentioned in issue #24027 (closed)
Milestone changed to %8.15
@markglenfletcher I think this will be fixed resolving this issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/17975
I'm not sure it's a good idea to export a decrypted version of the secrets file. We still have to figure out what's the best thing to do, but for now users can copy across the key from one instance to the new one, it could be through a safe SSH connection, etc...
I'm also not sure we want to ship an export or backup with everything decrypted so it can be encrypted again with a new key. I suggest we discuss other solutions here: https://gitlab.com/gitlab-org/gitlab-ce/issues/17975 - where there are pointers to some workarounds, such as just copying the old key.
Mentioned in issue #25121 (moved)
removed regression label
changed milestone to %8.16
added Deliverable label
Proposed solution https://gitlab.com/gitlab-org/gitlab-ce/issues/17975#note_20917347
changed milestone to %8.17
@jameslopez Could we simply not export encrypted data, at least for now, and mention that in the UI?
assigned to @jameslopez
mentioned in merge request !8739 (merged)
closed via merge request !8739 (merged)
mentioned in commit 4468104f
mentioned in merge request !14785
