Error 500 for a specific project

Hi,

We get "error 500" in each time we open the project https://git.vcpr.org/group-nsec/puppet with a web browser. The logs:

Started GET "/group-nsec/puppet" for 10.137.16.166 at 2016-08-10 17:38:06 +0000
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"group-nsec", "id"=>"puppet"}
Completed 500 Internal Server Error in 33ms (ActiveRecord: 3.6ms)

OpenSSL::Cipher::CipherError (bad decrypt):
  app/models/project.rb:481:in `import_url'
  app/models/project.rb:517:in `external_import?'
  app/models/project.rb:509:in `import?'
  app/models/project.rb:525:in `import_in_progress?'
  app/controllers/projects_controller.rb:94:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

All other projects work very well, except this one.

Any idea how to solve this issue?

Thank you.

Asher256

Hey @asher256, have you recently moved to a new server and restored your GitLab instance backup there?

Hello Mark. YES, I restored Gitlab from a backup.

@asher256 We've seen this before and I believe it's caused by not migrating the gitlab-secrets.json file over to the new server upon which you are running the restore. Please see the following documentation: http://docs.gitlab.com/ce/raketasks/backup_restore.html#prerequisites

If you have a backup of that file can you try following the steps in the documentation to migrate it to the new server?

But all other projects work very well. I have the issue on one project only.

I don't have any backup for gitlab-secrets.json. Is there any other solution?

Thank you!

Asher256

Hi @asher256, thanks for the extra information. Can you verify that the error occurs when trying to load the ProjectImportData for this project?

Start a rails console:

sudo gitlab-rails console

Find the project and attempt to load the ProjectImportData for it:

p = Project.find_with_namespace('group-nsec/puppet')
p.import_data

Thanks! I'll use the command you sent me.

@asher256 Did you get the chance to try this out?

Hi Mark,

I didn't have a chance to do try "p.import_data" because I had to find a quick and dirty solution for the group-nsec team. I deleted + recreated the project to solve the issue before your answer :( .

Can you give me extra information about p.import_data? (I will try it if I have the issue "Error 500" again)

Thank you, Mark!

The import_data method loads the ProjectImportData associated with the project. Which will also attempt to decrypt the attr_encrypted credentials and should cause the same error from the command line. I think we'll need to delete the ProjectImportData to be rid of the 500 error.

How can I delete the ProjectImportData? Is it safe? (no loss of data)

First of all, was the error reproducible by running the above commands?

Added backup-restore label

After the deletion + creation of the repository, I tried your command and I didn't give me any error.

If you deleted the Project, the associated import data would also be deleted. You are now no longer seeing the 500 error for that project?

No, I'm no longer seeing the 500 error for that project.

@asher256 Thanks. I'll close this down for now then.

Status changed to closed

Mentioned in issue #24458 (closed)

@markglenfletcher Could you suggest the .secret file location for 9.* ? I'm experience the same problem but can't find .secret file

@yuriypadlyak Please add more information about your environment. Are you using omnibus or did you install from source?

@markglenfletcher It was 10.0 Omnibus install backup migrated from 10.0 source install.

I've resolved it already by copying db_key_base from ~/gitlab/config/secrets.yml of the custom install to the /etc/gitlab/gitlab-secrets.json of the Omnibus. I've missed the info about importance of backing secrets up along with the main backup file. Also, it is in a different format, so I wounder how I should have converted the secrets for Omnibus.

Hello,

I too have just encountered this error. gitlab CE Omnibus 9.5.5

Only on two projects the 500 error occurs when attempting to access them via the web page. As the project owner is away this week I cannot confirm if the affected project was imported from another as some suggest is the cause.

I tried the rails console suggestion from earlier in this thread to see if the error occurs relating to the decryption: p = Project.find_in_namespace('a/b') but it says no such method and suggested use in_namespace. Trying p = Project.in_namespace('a/b') returned something but p.import_data is again no such method. is version 9.5 not using those methods any longer.

Is there an updated way to do that test? In any event is the troubleshooting the same? Instructions refer to removing ProjectImportData. Are these instructions available?

I looked at db_key_base currently in /etc/gitlab/gitlab-secrets.json and it is the same value as a backup taken of that file about 4 weeks ago. I believe (again the user is away this week) that they were demonstrating using the relevant data last week so perhaps there is an issue unrelated to db_key_base.

Either way, any help you can offer will be much appreciated.

thanks.

r.