Post Upgrade, One Projects Root Page is not loading

I'm glad it's not just me getting this. I am getting the same error/symptoms as well, though I am not sure if it's related to a merge request.

Same problem for me since upgrade to 8.9.0-ce.0

It appears I also have the same issue with some projects without any pending merge requests. This might actually not be related with MR ...

same problem for me after upgrade to 8.9.0

This dirty fix removes this error, but I don't know which other GitLab parts it will break, so use it on your own risk :)

--- project.old 2016-06-24 10:29:02.289882631 +0300
+++ project.rb  2016-06-24 10:30:29.329023802 +0300
@@ -453,8 +453,9 @@ class Project < ActiveRecord::Base

   def import_url
     if import_data && super
-      import_url = Gitlab::UrlSanitizer.new(super, credentials: import_data.credentials)
-      import_url.full_url
+#      import_url = Gitlab::UrlSanitizer.new(super, credentials: import_data.credentials)
+#      import_url.full_url
+      super
     else
       super
     end

I see this behaviour with a slightly different message

ArgumentError (key must be 32 bytes or longer):
  app/models/project.rb:456:in `import_url'
  app/models/project.rb:486:in `external_import?'
  app/models/project.rb:478:in `import?'
  app/models/project.rb:494:in `import_in_progress?'
  app/controllers/projects_controller.rb:93:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

Same error as @KingCrunch. I'm using GitLab 8.9.0 (12d9e057), and only one of my projects has this behaviour.
Did some tests, it appears that import_data doesn't has credentials.

It was my first time writing Ruby code, but It works with a fix similar to @yakovenkov's.
I just changed the if condition to:

if import_data && super && import_data.has_attribute?('credentials')

To be honest, it doesn't feel right to do so.
import_data has other attributes related to credentials, but not credentials:

encrypted_credentials
encrypted_credentials_iv
encrypted_credentials_salt

Hope it helps!

I have the same error using docker-compose, image '8.9.0-ce.0'. This is effecting multiple projects that I have.

I can confirm that this issue is fixed with the patching of line 455 of the file /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb described by @thequestion . I needed to restart GitLab (gitlab-ctl restart) to apply the change - not sure if there is a smarter way to clean the Rails (?) cache. But it works for me.

mentioned in issue #19125 (closed)

I can confirm as well that the issue is resolved with the proposed change by @thequestion and a subsequent gitlab-ctl restart

will update this thread if I hit any side effects. thanks all!

Thanks for checking this! It's strange that this happened after this release, I think the only recent change was the upgrade of attr_encrypted. It would be interesting to know if the following attributes are actually NULL in the database:

encrypted_credentials
encrypted_credentials_iv
encrypted_credentials_salt

Reassigned to @jameslopez

@jameslopez In my case, they aren't NULL. The only NULL field is data (in project_import_data table).
It seems that there is a CR at the end of encrypted_credentials and encrypted_credentials_iv, but not encrypted_credentials_salt.

@jameslopez You can set the attribute to "insecure mode" to prevent the key length error. That may help, although I generally would recommend decrypting and re-encrypting with a stronger key.

I wonder where the import method is getting the secret key from?

@connorshea don't think that's an option. There must be a logical explanation as to what has changed for this to occur.

It's a shared key Gitlab::Application.secrets.db_key_base which I believe is different per GitLab instance. Not sure if there were changes to that recently either, but it shouldn't change at all for an upgrade. I will investigate more tomorrow morning - it's quite late here now

My GITLAB_SECRETS_DB_KEY_BASE variable eventually and historically is shorter than 32 symbols. Is there any way to change it without loosing the data? What data will I loose if I change it manually?

attr_encryptedversion is set to 1.3.4 is 8.8's gemfile, when the version is 3.0.0 for 8.9. They seems to have added a check about key length ( https://github.com/attr-encrypted/encryptor#options , https://github.com/attr-encrypted/encryptor/issues/21 ), an since gitlab suggest at least 30 in the configuration's file, some db_key_base are also less than 32 characters (including mine - as reported in #19125 (closed) I have some broken projects and cannot create new one :( ).

I tried to find a way to change the db_key_base to 32 bytes to confirm it's the problem, but I didn't find a way without loosing the data :(

Ok did a little bit more research and using insecure_mode is actually going back to where we were before the upgrade, it just adds some length checks here https://github.com/attr-encrypted/encryptor/blob/master/lib/encryptor.rb#L59

This was definitely something we missed when doing the upgrade as in the legacy test that attr_encrypted used to test backward-compatibility the option insecure_mode is set - https://github.com/attr-encrypted/attr_encrypted/blob/master/test/legacy_compatibility_test.rb

Obviously as @connorshea suggested we could decrypt and encrypt everything again making sure we now use a key of longer length - we might as well change the algorithm to something more secure at the same time. See this comment:

The dynamic finders available in ActiveRecord will only work with :single_iv_and_salt mode. It is strongly advised that you do not use this mode. If you can search the encrypted data, it wasn't encrypted securely. This functionality will be deprecated in the next major release. quoted from https://github.com/attr-encrypted/attr_encrypted#Deprecations

I would suggest to apply the easiest solution first. That would be to use insecure_mode which is basically what we had until 8.9, and that should fix this regression. We can then create a technical debt issue to fix this perhaps for 8.10 which will have to do at the very least the following:

Change https://gitlab.com/gitlab-org/gitlab-ci/blob/master/config/secrets.yml.example - wrong advise as minimum is 32:

OpenSSL::Cipher.new('aes-256-cbc').key_len
> 32

Decrypt and encrypt the following:

File	Field
user.rb	otp_secret
project_import_data.rb	credentials
variable.rb	value
pages_domain.rb	key

Check oauth_session.rb and the implications of the change - we are also using db_key_base there.
Generate new db_key_base and update the GitLab config files - possibly /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json for omnibus. But if this is set in the environment we may need to output the new key so people can set this manually as an ENV variable.
Create a migration to decrypt the data before the db_key_base has changed and do the update below and encrypt the data again...

And probably more things that are missing. I think there are quite a few things that can go wrong here so I'm a bit reluctant to do this for 8.9.x

Waiting for your thoughts on this @connorshea / @DouweM / @rspeicher

Milestone changed to %8.9

Added ~12980 regression labels

@the-glu that's correct. In order to not to lose the data we would need to migrate the data as explained in my last comment. Now if you are using CE and not EE, or using EE with no mirrors set, you could try the following:

sudo gitlab-rails runner  "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"

Basically deleting all import data. In theory those credentials are no longer needed once the project has been imported successfully. This would prevent the 500 error in the homepage.

@jameslopez thanks! I also commented the block on line 24 in vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb and now I can also create projects again :)

(This is a very ugly and bad fix, I know ^^')

@jameslopez sounds good to me, however my understanding is that it will require taking each instance offline in order to decrypt and then re-encrypt everything. While we're at it, we should re-encrypt them with aes-128-gcm if possible. I'm not sure what version of OpenSSL is necessary to support that cipher, but they shouldn't be using outdated OpenSSL versions anyway.

This is discussed more in my issue about the gem update #17145 (closed).

@the-glu it's probably a very bad idea to mess with the encryption library.

I'm getting similar things.

on a ci build run i get following:

OpenSSL::Cipher::CipherError (bad decrypt): /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in final' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in crypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:233:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:317:in decrypt' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-3.0.1/lib/attr_encrypted.rb:155:in block (2 levels) in attr_encrypted' /home/git/gitlab/app/models/ci/build.rb:412:in block in project_variables' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in map' /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:46:in map' /home/git/gitlab/app/models/ci/build.rb:411:in project_variables' /home/git/gitlab/app/models/ci/build.rb:122:in `variables'

...

its very long, tell me if you need all of this.

if i want view the "variables" page of this project i get following:

Completed 500 Internal Server Error in 143ms (ActiveRecord: 9.1ms)

ActionView::Template::Error (bad decrypt): 13: - if variable.id? 14: %tr 15: %td= variable.key 16: %td= variable.value 17: %td 18: = link_to namespace_project_variable_path(@project.namespace, @Project, variable), class: "btn btn-transparent btn-variable-edit" do 19: %span.sr-only app/views/projects/variables/_table.html.haml:16:in block in _app_views_projects_variables__table_html_haml__3708802409549015508_75212180' app/views/projects/variables/_table.html.haml:12:in _app_views_projects_variables__table_html_haml__3708802409549015508_75212180' app/views/projects/variables/index.html.haml:17:in _app_views_projects_variables_index_html_haml___1311328960352402588_86617380' lib/gitlab/middleware/go.rb:16:in call'

my secret key is definitely to short, but i don't know how to correctly change it. and whats affected if i change it, and if there is some cleanup needed after this?

hopefully someone can help me out.

the project page itself runs after the fix from @yakovenkov

thx.

@seti that's definitely related to the same issue. We'll put a fix ASAP - for the next patch release hopefully.

There are a few things that can go wrong that I listed in the table of this comment https://gitlab.com/gitlab-org/gitlab-ce/issues/19073#note_12712458 and variables is one of them

mentioned in issue #19043 (closed)

@jameslopez thx for fast reply. is the insecure_mode fixing the problem temporarly? where to set it?

i don't know how to decrypt with old key ond encrypt with new key. but if it gets fixed in a hopfully near bugfix release i can wait.

but maybe its a good idea to implement a function to update the secret keys, isn't it?

it's a simple fix - just adding insecure_mode: true to the configuration hash passed to attr_encrypted on the files I've mentioned above. The number of files affected change between CE and EE. We should have this fixed very soon for 8.9.3 as I'll submit the fix first thing tomorrow morning, for both CE and EE :)

Updating the secret keys seems like the best option, but it could be a bit of a nasty migration. We'll probably do that next anyway.

mentioned in issue #18343 (closed)

mentioned in issue #19211 (closed)

@jameslopez, you mean like this (example variable.rb):

    attr_encrypted :value,
       mode: :per_attribute_iv_and_salt,
       key: Gitlab::Application.secrets.db_key_base,
       algorithm: 'aes-256-cbc',
       insecure_mode: true

Doing that for user.rb, project_import_data.rb, & variable.rb in CE did not fix this error:

tarted GET "/xyz/abc" for x.x.x.x at 2016-06-27 16:52:16 -0700
Processing by ProjectsController#show as HTML
Parameters: {"namespace_id"=>"xyz", "id"=>"abc"}
Completed 500 Internal Server Error in 93ms (ActiveRecord: 4.7ms)

ArgumentError (iv must be 16 bytes or longer):
app/models/project.rb:456:in `import_url'
app/models/project.rb:486:in `external_import?'
app/models/project.rb:478:in `import?'
app/models/project.rb:494:in `import_in_progress?'
app/controllers/projects_controller.rb:93:in `show'
lib/gitlab/middleware/go.rb:16:in `call'

I tried this for CE on-premise:

cd /home/git/gitlab
sudo -u git -H vim ./vendor/bundle/ruby/2.1.0/gems/encryptor-3.0.0/lib/encryptor.rb +21

update the line insecure_mode from false to true like this:

  # Run 'openssl list-cipher-commands' in your terminal to view a list all cipher algorithms that are supported on your platform
  def default_options
    @default_options ||= { algorithm: 'aes-256-gcm',
                           auth_data: '',
                           insecure_mode: true,
                           hmac_iterations: 2000,
                           v2_gcm_iv: false }
  end

Then, restart the gitlab service like sudo service gitlab restart. I had a problem when starting again, and executing again sudo service gitlab start all went ok:

$ sudo service gitlab stop
Shutting down GitLab Unicorn
Shutting down GitLab Sidekiq
Shutting down GitLab Workhorse
.
GitLab is not running.

$ sudo service gitlab start
Starting GitLab Unicorn
Starting GitLab Sidekiq
Starting GitLab Workhorse
master failed to start, check stderr log for details

The GitLab Unicorn web server is not running.
The GitLab Sidekiq job dispatcher with pid 18580 is running.
The GitLab Workhorse with pid 18563 is running.

$ sudo service gitlab start
Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran.
Starting GitLab Unicorn
The Sidekiq job dispatcher is already running with pid 18580, not restarting
The GitLab Workhorse is already running with pid 18580, not restarting

The GitLab Unicorn web server with pid 19932 is running.
The GitLab Sidekiq job dispatcher with pid 18580 is running.
The GitLab Workhorse with pid 18563 is running.
GitLab and all its components are up and running.

The problem shown in unicorn.stderr.log was:

I, [2016-06-28T08:50:51.817793 #18540]  INFO -- : Refreshing Gem list
/home/git/gitlab/lib/gitlab/popen.rb:29:in `value': No live threads left. Deadlock? (fatal)
	from /home/git/gitlab/lib/gitlab/popen.rb:29:in `block in popen'
	from /usr/local/lib/ruby/2.1.0/open3.rb:199:in `popen_run'
	from /usr/local/lib/ruby/2.1.0/open3.rb:93:in `popen3'
	from /home/git/gitlab/lib/gitlab/popen.rb:23:in `popen'
	from /home/git/gitlab/config/initializers/2_app.rb:7:in `<module:Gitlab>'
	from /home/git/gitlab/config/initializers/2_app.rb:1:in `<top (required)>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `block in load'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:652:in `block in load_config_initializer'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:651:in `load_config_initializer'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `each'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/engine.rb:615:in `block in <class:Engine>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:226:in `block in tsort_each'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:418:in `block (2 levels) in each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:427:in `each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:417:in `block in each_strongly_connected_component_from'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `each'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:44:in `tsort_each_child'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:411:in `call'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:411:in `each_strongly_connected_component_from'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:347:in `block in each_strongly_connected_component'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:345:in `each'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:345:in `call'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:345:in `each_strongly_connected_component'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:224:in `tsort_each'
	from /usr/local/lib/ruby/2.1.0/tsort.rb:205:in `tsort_each'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
	from /home/git/gitlab/config/environment.rb:5:in `<top (required)>'
	from config.ru:19:in `require'
	from config.ru:19:in `block in <main>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `instance_eval'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `initialize'
	from config.ru:1:in `new'
	from config.ru:1:in `<main>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn.rb:48:in `eval'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn.rb:48:in `block in builder'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn_rails:139:in `call'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn_rails:139:in `block in rails_builder'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:768:in `call'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:768:in `build_app!'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:137:in `start'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/unicorn-4.9.0/bin/unicorn_rails:209:in `<top (required)>'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/unicorn_rails:23:in `load'
	from /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/unicorn_rails:23:in `<main>'

And then, the pages that did not work previously, now they work. Thanks for the information.

Hoping this helps as a patch.

Jordi

@bbodenmiller I wonder if you restarted your instance after the change?

@jpahullo I'm glad it worked for you!

I'll submit a MR soon.

mentioned in issue #19218 (closed)

mentioned in merge request !4963 (merged)

Status changed to closed by merge request !4963 (merged)

mentioned in issue #19261 (closed)

This has now been fixed and should be released as part of 8.9.3

There's still a technical debt issue to be resolved as a consequence - to implement a better encrypting mode: https://gitlab.com/gitlab-org/gitlab-ce/issues/19261

mentioned in commit 2c3f3cb3

@jameslopez I tried a restart, no luck. Does insecure_mode: true need to come before algorithm:? If not, I've got the same thing as https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4963 but in a different order and still getting the same error. Is it possible this only fixes some of the errors reported in this issue?

@bbodenmiller that's a bit strange. The only place where you could get that error raised seems to be https://github.com/attr-encrypted/encryptor/blob/3.0.0/lib/encryptor.rb#L62 and that implies that insecure_mode is false. Order doesn't matter.

I would say you either haven't edited the right source file or the restart didn't work somehow... Are you using the omnibus package or is it an install from source ?

@jameslopez after applying your patch to 3 files a different kind of error appears at same places. It seems that OpenSSL::Cipher has some internal checks of IV.

Completed 500 Internal Server Error in 1307ms (ActiveRecord: 320.3ms)

OpenSSL::Cipher::CipherError (iv length too short):
  app/models/project.rb:456:in `import_url'
  app/models/project.rb:486:in `external_import?'
  app/models/project.rb:478:in `import?'
  app/models/project.rb:494:in `import_in_progress?'
  app/controllers/projects_controller.rb:93:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

@yakovenkov Have you done any changes to the key GITLAB_SECRETS_DB_KEY_BASE ? And do you know what its length is for your setup?

@jameslopez If I understand correctly the length of GITLAB_SECRETS_DB_KEY_BASE is the length of db_key_base field under gitlab_ci section in gitlab-secrets.json file. This string is 128 chars == 64 bytes long, I never changed it manually.

thanks @yakovenkov, I asked that as changing the key will definitely affect the iv field. Could you find out what the size of the fields encrypted_credentials, encrypted_credentials_iv, and encrypted_credentials_salt is?

Perhaps there's something else going on here though - I did try a few things locally but couldn't replicate the last error. Having a path to replicate it would help a lot!

@jameslopez Sorry, have no idea of how to determine sizes of this fields (I'm not a Ruby developer at all). I think I can setup a clone of our GitLab server and give you full access to it. If this solution will be suitable, please contact me via email yakovenkov at gmail dot com.

@yakovenkov are you running the omnibus version of GitLab or the installation from source ? Let's try to get the info from here first thanks!

@jameslopez When can we expect 8.9.3 to be released as stable?

@cparmar I would expect 8.9.3 to be released very soon. Perhaps today or tomorrow - you can follow the issue on https://gitlab.com/gitlab-org/gitlab-ce/issues/19245

applied v8.9.3 and all is ok by now. Thanks! I reverted also my change on the gem's code.

@jameslopez Upgrading to 8.9.3 doesn't help. I'm using Ubuntu/Trusty DEB packages downloaded from packages.gitlab.com It has upgraded DB, but nothing changed, the error remains the same:

Started GET "/xxx/xx" for 127.0.0.1 at 2016-06-30 10:04:21 +0300
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"xxx", "id"=>"xx"}
Completed 500 Internal Server Error in 286ms (ActiveRecord: 26.4ms)

OpenSSL::Cipher::CipherError (iv length too short):
  app/models/project.rb:456:in `import_url'
  app/models/project.rb:486:in `external_import?'
  app/models/project.rb:478:in `import?'
  app/models/project.rb:494:in `import_in_progress?'
  app/controllers/projects_controller.rb:93:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

I tried to add a line of code to view encrypted_credentials_iv etc (line 456 of project.rb)

Rails.logger.info "1: #{@import_data.encrypted_credentials}"

But got this in production.log:

NoMethodError (undefined method `encrypted_credentials' for nil:NilClass):

Moreover, it seems that import_data is nil itself.

@yakovenkov I see. It does look like a different problem. Can you try changing that line to:

 Rails.logger.info "1: #{import_data.inspect}"

Feel free to obfuscate the values a bit replacing a couple of characters if you want.

@jameslopez Here it is:

1: #<ProjectImportData id: 1, project_id: 2, data: nil, encrypted_credentials: "EHj7YTafa6/XPdg3Fjew+OrDkSrn1OrM3bYZs7qPpX529mTLwQ...", encrypted_credentials_iv: "ECqMo/I6ULzockqM ", encrypted_credentials_salt: "_hUttFZNAkrnJm4wyM4tFSA== ">
Completed 500 Internal Server Error in 362ms (ActiveRecord: 26.2ms)

OpenSSL::Cipher::CipherError (iv length too short):
.....

Many thanks @yakovenkov. Ok... Very strange. Can I suggest the following? Guessing a bit but I need to you run this in the command line:

 sudo gitlab-rails runner "ProjectImportData.find(1).encrypted_credentials_iv.strip!"

Basically I was expecting encrypted_credentials_iv not to have that final trailing whitespace.

@jameslopez Nothing has changed, it prints encrypted_credentials_iv with trailing whitespace as before.

I do a little investigation of the problem, here's my mention of it (for project_import_data.rb).

It uses attr_encrypted class and sets per_attribute_iv_and_salt mode. At the attr_encrypted.rb, line 361 it calls load_iv_for_attribute routine, which does:

options[:encode_iv] = options[:default_encoding] if options[:encode_iv] == true
...
iv = iv.unpack(encode_iv).first if encode_iv
...

So, it assumes that encrypted_credentials_iv is base64 encoded. In my case, encrypted_credentials_iv equals to 'ECqMo/I6ULzockqM ', which is invalid Base64 data with last whitespace. If I remove it, the remaining string decodes to 12 bytes but not to 16 bytes! And that is why OpenSSL routines declines to use it.

So, the problem is in incorrect data in database table. I don't know which value should be there so I simply clear the project_import_data table and all is working fine now.

great @yakovenkov, deleting the import data should be fine anyway. I believe something went wrong with that data in any case

@jameslopez I have just updated to 8.9.4 but I am having the same issue as @yakovenkov.

Can I get some advice on how to "clear the project_import_data table" please.

@cparmar see this comment - https://gitlab.com/gitlab-org/gitlab-ce/issues/19073#note_12712670

That will work if you are on CE or in EE with no mirrored repos.

Thanks @jameslopez, that's done the trick.

@jameslopez I'm running from source and don't have gitlab-runner. Is the following statement equivalent (ie: should I run the following)?

~~sudo -u git -H bundle exec rails runner "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }" production~~

Edit

Nevermind, I ran it slightly altered and that seemed to do the trick:

sudo -u git -H bundle exec rails runner -e production "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"

Same problem for random projects after updated to 8.9.4 gitlab: the command: (sudo gitlab-rails runner "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }") fixed the issue

==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx6 - - [06/Jul/2016:07:43:14 -0400] "POST /ci/api/v1/builds/register.json HTTP/1.1" 404 27 "-" "gitlab-ci-multi-runner 1.3.2 (stable-1-3; go1.6.2; linux/amd64)"

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/hubpen/hubpen" for 8xxx at 2016-07-06 07:43:14 -0400
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"hubpen", "id"=>"hubpen"}
Completed 500 Internal Server Error in 159ms (ActiveRecord: 7.3ms)

OpenSSL::Cipher::CipherError (iv length too short):
  app/models/project.rb:456:in `import_url'
  app/models/project.rb:486:in `external_import?'
  app/models/project.rb:478:in `import?'
  app/models/project.rb:494:in `import_in_progress?'
  app/controllers/projects_controller.rb:93:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'



==> /var/log/gitlab/gitlab-workhorse/current <==
2016-07-06_11:43:15.07925 2016/07/06 07:43:15 ErrorPage: serving predefined error page: 500

thanks @AngeIII - a bit too late now, but in order to submit a fix it will help if someone could send me the value of encrypted_credentials, encrypted_credentials_iv, encrypted_credentials_saltso I can investigate further :) Definitely still related to us upgrading attr_encrypted

mentioned in issue #19755 (closed)

Same here,gitlab-ce 8.9.6-ce.0

Started GET "/deb/deb" for at 2016-07-14 14:12:07 +0300 Processing by ProjectsController#show as HTML Parameters: {"namespace_id"=>"deb", "id"=>"deb"} Completed 500 Internal Server Error in 68ms (ActiveRecord: 5.1ms)

OpenSSL::Cipher::CipherError (iv length too short): app/models/project.rb:456:in import_url' app/models/project.rb:486:in external_import?' app/models/project.rb:478:in import?' app/models/project.rb:494:in import_in_progress?' app/controllers/projects_controller.rb:93:in show' lib/gitlab/middleware/go.rb:16:in call'

Status changed to reopened

@anton.iutin (or anyone else with the problem) would you be able to send me the following details? Please do not paste it here but send it to my e-mail.

Assuming your project is called deb under the namespace deb, if you installed GitLab from source:

sudo -u git -H bundle exec rails runner -e production  "p Project.find_with_namespace('deb/deb').import_data.inspect; p Gitlab::Application.secrets.db_key_base"

If you installed GitLab using the omnibus packages:

 sudo gitlab-rails runner "p Project.find_with_namespace('deb/deb').import_data.inspect; p Gitlab::Application.secrets.db_key_base"

This would be very helpful as we don't have this issue at gitlab.com so it's not easy to replicate...

Many thanks!

Milestone changed to %8.10

mentioned in issue #19857 (closed)

mentioned in issue #19025 (closed)

After a few tests locally, I suspect this has something to do with the migration related to import_data. Very difficult to replicate though!

@jameslopez Do you still want data? I'm seeing this in 8.9.6 after upgrading sunday.

@dstanley that will definitely help, yes! See my comment https://gitlab.com/gitlab-org/gitlab-ce/issues/19073#note_13065124

Definitely a problem with the migration. All my projects that existed before the upgrade have the shorter (16-char) iv string, while new projects created after the upgrade have the longer (25-char) iv string, including the ==+ at the end.

Looking at the difference between 20160302152808_remove_wrong_import_url_from_projects.rb and app/models/project_import_data.rb I see that in app/models/project_import_data.rb you specify "algorithm: 'aes-256-cbc'", while in the migration you do not specify the algorithm. Looking at attr_encrypted, it seems the default algorithm is 'aes-256-gcm'. Perhaps this is the source of the problem?

mentioned in issue #19930 (closed)

@dstanley that has something to do yep What's actually happening:

GitLab old version	GitLab upgraded version	Default algorithm	Key length restriction
8.6 or earlier	8.7	aes-256-cbc	No
8.6 or earlier	8.8	aes-256-cbc	No
8.6 or earlier	8.9	aes-256-gcm	Yes

I see 2 different issues, not only the algorithm but the migration may encounter problems if the key is too short. My understanding is that this is affecting upgrades from GitLab version 8.6 or earlier to 8.9 (and up to 8.10RC11).

My actions now would be to provide a fix for 8.10 so anyone that runs that migration to upgrade from GitLab <= 8.6 to 8.10 won't encounter that problem. I will also provide a better workaround for those with the problem - although deleting the import_data no longer needed is fine.

Something to note is that the original issue with the length problem ArgumentError (iv must be 16 bytes or longer) (different from OpenSSL::Cipher::CipherError (iv length too short)) should be fixed already.

Status changed to closed by commit grzesiek/gitlab-ce@2c3f3cb3920c5242cbc66456b3eed423636a2128

Status changed to reopened

mentioned in merge request !5358 (merged)

Created https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5358 that should fix this issue for upgrades to 8.10

Also, I added a new workaround (haven't tested it thoroughly) for people still encountering this problem that does not remove the import data:

bash <(curl -s --GET https://gitlab.com/snippets/22783/raw | tr -d '\015')

That will run a bash script that also runs the following snippet https://gitlab.com/snippets/22772 which transforms and fixes the import_data that have the OpenSSL::Cipher::CipherError (iv length too short) problem. It should work for both GitLab installed with omnibus and installation from source.

mentioned in issue #19582 (closed)

Status changed to closed by merge request !5358 (merged)

mentioned in commit 982d27d1

mentioned in commit ab883cf9

Hello, I had gitlab omni package Just migrated from gitlab-ce-7.12.2 to gitlab-ce-8.9.6. Update returned success but I'm receiving Error 500 in one of the projects. Looking into the log I see:

==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/developers/facestore" for 193.137.26.66 at 2016-07-20 15:37:47 +0100 Processing by ProjectsController#show as HTML Parameters: {"namespace_id"=>"developers", "id"=>"facestore"} Completed 500 Internal Server Error in 98ms (ActiveRecord: 4.3ms)

OpenSSL::Cipher::CipherError (iv length too short): app/models/project.rb:456:in import_url' app/models/project.rb:486:in external_import?' app/models/project.rb:478:in import?' app/models/project.rb:494:in import_in_progress?' app/controllers/projects_controller.rb:93:in show' lib/gitlab/middleware/go.rb:16:in call'

Tried to execute:

bash <(curl -s --GET https://gitlab.com/snippets/22783/raw | tr -d '\015')

But receive error:

[root@gitlab ~]# bash <(curl -s --GET https://gitlab.com/snippets/22783/raw | tr -d '\015') /usr/bin/gitlab-rails /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62:in eval': /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62: unknown regexp options - tp (SyntaxError) from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62:in <top (required)>' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:123:in require' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:123:in require_command!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:90:in runner' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in run_command!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in <top (required)>' from bin/rails:9:in require' from bin/rails:9:in `'

Anyone can help?

Thanks in advance.

Bruno Morais

@MrAbaddon could you try again?

I haven't had a chance to test this much on omnibus, unfortunately. So I really appreciate your help!

Hi James, Thanks for the reply. I've tried running several times the command. I've updated my Linux distro (CentOS release 6.8 (Final)) and runned again with no luck. Do you need any more info?

@MrAbaddon I've updated the script behind the command just a minute ago. Does the output/error changed at all ?

Returned a diferent error:

[root@gitlab ~]# bash <(curl -s --GET https://gitlab.com/snippets/22783/raw | tr -d '\015') /usr/bin/gitlab-rails /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in require': cannot load such file -- /tmp/tmp.5Dyhu7TCxX/import_data_fix.rb (LoadError) from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in block in require' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in load_dependency' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in require' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62:in <top (required)>' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62:in eval' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/runner.rb:62:in <top (required)>' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:123:in require' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:123:in require_command!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:90:in runner' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in run_command!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in <top (required)>' from bin/rails:9:in require' from bin/rails:9:in '

@MrAbaddon ah, strange, as I have tested it in another omnibus installation and seemed to work. Changed a couple of things, could you try again ?

@jameslopez Worked like a charm... :)

Thanks for the help. All projects working.

Script with that where clause didnt worked for me I had to:

class ProjectImportDataFake
  extend AttrEncrypted
  attr_accessor :credentials
  attr_encrypted :credentials, key: Gitlab::Application.secrets.db_key_base, marshal: true, encode: true, :mode => :per_attribute_iv_and_salt

  def initialize(encrypted_credentials, iv, salt)
    @encrypted_credentials = encrypted_credentials
    @encrypted_credentials_iv = iv
    @encrypted_credentials_salt = salt
  end
end



for project in Project.all
  id = project.import_data
  begin
    id.credentials
  rescue OpenSSL::Cipher::CipherError => error
    new_import_data = ProjectImportDataFake.new(id.encrypted_credentials,
                                                id.encrypted_credentials_iv,
                                                id.encrypted_credentials_salt)
    puts "#{project.name} #{error} #{new_import_data.credentials}"
    id.credentials = new_import_data.credentials
    id.save
  end if id != nil
end and nil

I was unable to get the bash command to run with the Omnibus install but I was able to take the code from the Snippet and run it in sudo gitlab-rails console to fix my issues.

@thequestion 's solution works for me.

Mentioned in commit pfjason/gitlab-ce@ab883cf9

Mentioned in commit pfjason/gitlab-ce@106f7f30

Mentioned in commit pfjason/gitlab-ce@2c3f3cb3

I'm still having trouble with this, after upgrading to 9.0.0

I've tried running the script at https://gitlab.com/snippets/22772/raw in the gitlab-rails cosnsole as user root and user git, but always get:

irb(main):107:0* class ProjectImportDataFake
attr_accessor :credentials
irb(main):108:1> extend AttrEncrypted
irb(main):109:1> attr_accessor :credentials
irb(main):110:1> attr_encrypted :credentials, key: Gitlab::Application.secrets.db_key_base, marshal: true, encode: tru e, :mode => :per_attribute_iv_and_salt
irb(main):111:1>
end
irb(main):112:1* def initialize(encrypted_credentials, iv, salt)
irb(main):113:2> @encrypted_credentials = encrypted_credentials
irb(main):114:2> @encrypted_credentials_iv = iv
irb(main):115:2> @encrypted_credentials_salt = salt
irb(main):116:2> end
irb(main):117:1> end
=> :initialize

irb(main):118:0>
ProjectImportData.where("encrypted_credentials_iv like '% '").each do |import_data|
old_import_data = ProjectImportDataFake.new(import_data.encrypted_credentials,
irb(main):119:0* ProjectImportData.where("encrypted_credentials_iv like '% '").each do |import_data|
irb(main):120:1* old_import_data = ProjectImportDataFake.new(import_data.encrypted_credentials,
puts 'Done.'irb(main):121:2* import_data.encrypted_credentials_iv,
irb(main):122:2* import_data.encrypted_credentials_salt)
irb(main):123:1>
irb(main):124:1* import_data.credentials = old_import_data.credentials
irb(main):125:1> import_data.save!
irb(main):126:1> end
OpenSSL::Cipher::CipherError: iv length too short
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:79:in iv=' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:79:in crypt'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in decrypt' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:233:in decrypt'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:317:in decrypt' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:155:in block (2 l evels) in attr_encrypted'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_recor d.rb:75:in block in attr_encrypted' from (irb):124:in block in irb_binding'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:4 6:in each' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:4 6:in each' from (irb):119
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:110:in start' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/console.rb:9:in start'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:68:in co nsole' from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:39:in ru n_command!'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/commands.rb:17:in <top (required)>' from bin/rails:9:in require'
from bin/rails:9:in `'
irb(main):127:0>
irb(main):128:0* puts 'Done.'
Done.
=> nil

I also tried iianox's script with the same result.

I also tried both:

bash <(curl -s --GET https://gitlab.com/snippets/22783/raw | tr -d '\015')

and @aiionx's script.

I get, I think, the same error as @john.ryan :

openintegritytheme iv length too short {:user=>"XXXX", :password=>nil}
/home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:79:in `iv=': iv length too short (OpenSSL::Cipher::CipherError)
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:79:in `crypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:233:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:317:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted.rb:155:in `block (2 levels) in attr_encrypted'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:75:in `block in attr_encrypted'
        from iv.rb:26:in `rescue in block in <top (required)>'
        from iv.rb:19:in `block in <top (required)>'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:46:in `each'
        from iv.rb:17:in `<top (required)>'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/runner.rb:60:in `load'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/runner.rb:60:in `<top (required)>'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:123:in `require'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:90:in `runner'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
        from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.8/lib/rails/commands.rb:17:in `<top (required)>'
        from bin/rails:9:in `require'
        from bin/rails:9:in `<main>'

I would really appreciate a suggestion on how to fix this!

Thanks,

Jun

mentioned in merge request !14785

Post Upgrade, One Projects Root Page is not loading

Designs

Child items ...

Activity

on a ci build run i get following:

if i want view the "variables" page of this project i get following:

Admin message

Admin message

Post Upgrade, One Projects Root Page is not loading

Activity

on a ci build run i get following:

if i want view the "variables" page of this project i get following: