Skip to content
Snippets Groups Projects
New issue look: Off

Importing from gitlab.com to GitLab EE fails

    • Closed Issue created by username-removed-238606 @gajus

    Hi, I am trying out the GitLab EE.

    How do I migrate projects from gitlab.com to GitLab EE instance?

    I have done all the steps here http://docs.gitlab.com/ce/integration/gitlab.html

    Then tried migrating individual projects, but all migrations end with "failed" status.

    Once "migrated", repository has files but no commits and no issues.

    I have looked through couple of logs but I cannot find any obvious errors.

    In /var/log/gitlab/gitlab-rails/sidekiq.log logs I can see:

    2a0-4a2e-8013-202147a72a8e", "queue_name"=>"mailers", "arguments"=>["Notify", "group_access_granted_email", "deliver_now", 2], "locale"=>"en"}], "retry"=>true, "jid"=>"82b2c875aeb92ad460c37a2b", "created_at"=>1463239822.0083199, "enqueued_at"=>1463239822.0089939, "error_message"=>"Couldn't find GroupMember with 'id'=2 [WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = ?]", "error_class"=>"ActiveRecord::RecordNotFound", "failed_at"=>1463239822.0158296, "retry_count"=>0} 2016-05-14_15:30:22.01676 2016-05-14T15:30:22.016Z 14613 TID-v9i6g WARN: ActiveRecord::RecordNotFound: Couldn't find GroupMember with 'id'=2 [WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = ?]

    Screen_Shot_2016-05-14_at_16.37.20

    Designs

      Child items ...

      2. Activity

      • username-removed-238606 Title changed from Importing gitlab.com fails to Importing from gitlab.com to GitLab EE fails

        Title changed from Importing gitlab.com fails to Importing from gitlab.com to GitLab EE fails

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        I have captured all the logs sidekiq.log that appear during the importation process.

        2016-05-14_15:43:10.94333 2016-05-14T15:43:10.942Z 14613 TID-16fs50 RepositoryImportWorker JID-753ee11295c1b65f52b0ef8b INFO: start
2016-05-14_15:43:13.57702 2016-05-14T15:43:13.571Z 14613 TID-16fs50 RepositoryImportWorker JID-753ee11295c1b65f52b0ef8b INFO: fail: 2.629 sec
2016-05-14_15:43:13.57705 2016-05-14T15:43:13.571Z 14613 TID-16fs50 WARN: {"class"=>"RepositoryImportWorker", "args"=>[5], "retry"=>true, "queue"=>"gitlab_shell", "jid"=>"753ee11295c1b65f52b0ef8b", "created_at"=>1463240590.9406607, "enqueued_at"=>1463240590.9417632, "error_message"=>"undefined local variable or method `import_data' for #<Gitlab::GitlabImport::Importer:0x00000007c8e8b0>", "error_class"=>"NameError", "failed_at"=>1463240593.5713751, "retry_count"=>0}
2016-05-14_15:43:13.57705 2016-05-14T15:43:13.572Z 14613 TID-16fs50 WARN: NameError: undefined local variable or method `import_data' for #<Gitlab::GitlabImport::Importer:0x00000007c8e8b0>
2016-05-14_15:43:13.57706 2016-05-14T15:43:13.572Z 14613 TID-16fs50 WARN: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitlab_import/importer.rb:8:in `initialize'
2016-05-14_15:43:13.57706 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:62:in `new'
2016-05-14_15:43:13.57706 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:62:in `importer'
2016-05-14_15:43:13.57706 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:51:in `import_data'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gitlab-rails/app/services/projects/import_service.rb:23:in `execute'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gitlab-rails/app/workers/repository_import_worker.rb:13:in `perform'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
2016-05-14_15:43:13.57707 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call'
2016-05-14_15:43:13.57708 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-05-14_15:43:13.57708 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
2016-05-14_15:43:13.57708 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-05-14_15:43:13.57708 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
2016-05-14_15:43:13.57708 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/logging.rb:30:in `with_context'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/server/logging.rb:7:in `call'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `call'
2016-05-14_15:43:13.57709 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:132:in `invoke'
2016-05-14_15:43:13.57710 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:127:in `block in process'
2016-05-14_15:43:13.57710 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:166:in `stats'
2016-05-14_15:43:13.57711 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:126:in `process'
2016-05-14_15:43:13.57711 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:79:in `process_one'
2016-05-14_15:43:13.57712 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:67:in `run'
2016-05-14_15:43:13.57712 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:16:in `watchdog'
2016-05-14_15:43:13.57712 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/util.rb:24:in `block in safe_thread'

2016-05-14_15:44:09.00837 2016-05-14T15:44:09.007Z 14613 TID-osicaj734 RepositoryImportWorker JID-753ee11295c1b65f52b0ef8b INFO: start
2016-05-14_15:44:09.80590 2016-05-14T15:44:09.805Z 14613 TID-osicaj734 RepositoryImportWorker JID-753ee11295c1b65f52b0ef8b INFO: done: 0.798 sec
      • username-removed-238606 mentioned in issue #451 (closed)

        mentioned in issue #451 (closed)

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        This has been open now for 3 days.

        Is there an alternative channel for EE support?

      • Stan Hu
        Stan Hu @stanhu ·
        Maintainer

        Sorry for the delay. @jameslopez, can you take a look at this? Is import_data is defined for Gitlab::GitlabImport::Importer? Should it be project.import_data?

        /cc: @balameb

        Edited by Stan Hu
      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        @stanhu thanks for the heads up, I'll create a MR quickly.

      • James Lopez Reassigned to @jameslopez

        Reassigned to @jameslopez

      • James Lopez Milestone changed to 8.7

        Milestone changed to 8.7

      • James Lopez Added regression label

        Added regression label

      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        Added MR on CE gitlab-org/gitlab-ce!4181

      • Robert Speicher Status changed to closed by merge request gitlab-org/gitlab-ce!4181

        Status changed to closed by merge request gitlab-org/gitlab-ce!4181

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        Still does not work. (Exact same behaviour as described in the original issue.)

        Even worse, the log shows success:

        2016-05-24_22:08:21.66378 2016-05-24T22:08:21.662Z 7484 TID-es18o RepositoryImportWorker JID-5690960e10360515d205de7c INFO: start
2016-05-24_22:08:24.20021 2016-05-24T22:08:24.199Z 7484 TID-es18o RepositoryImportWorker JID-5690960e10360515d205de7c INFO: done: 2.537 sec

        While the UI says "failed".

        Edited by username-removed-238606
      • username-removed-238606 Status changed to reopened

        Status changed to reopened

      • username-removed-444214
        username-removed-444214 @MrChrisW ·
        Contributor

        Reported another user: https://gitlab.zendesk.com/agent/tickets/24395

      • Stan Hu
        Stan Hu @stanhu ·
        Maintainer

        This may be related to !416 (merged). @jameslopez, can you confirm?

      • username-removed-444214 Added ~113221 label

        Added ~113221 label

      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        @stanhu most likely, yes.

        @gajus is that using Gitlab EE ?

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        @gajus is that using Gitlab EE ?

        Yes. This information is given in the issue title and issue body, though.

      • Stan Hu
        Stan Hu @stanhu ·
        Maintainer

        @jameslopez I also saw a support issue where it appears the RepositoryImportWorker ran successfully but for some reason the directory didn't exist afterwards. When I looked, the repository exists? was true even though the directory was not present. I'm not quite sure how could that happen unless the directory got renamed from underneath or git clone quietly failed.

        Edited by Stan Hu
      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        thanks @gajus - looks like the info has been updated...

        @stanhu that seems weird. Can you link me to the support issue ? Thanks!

      • Stan Hu
        Stan Hu @stanhu ·
        Maintainer

        @jameslopez See gitlab-com/support-forum#732 (closed).

      • Robert Speicher Status changed to closed by commit dblessing/gitlab-ee@98d88e8188f551533390ca9dcb1247e5af6807cf

        Status changed to closed by commit dblessing/gitlab-ee@98d88e8188f551533390ca9dcb1247e5af6807cf

      • Stan Hu Status changed to reopened

        Status changed to reopened

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        Is there a way to do the migration at the moment? (gitlab.com to EE?)

      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        @gajus what version of GitLab EE are you running locally? Can you try with the latest 8.8.2 ? Thanks!

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author 
        2016-05-25_21:31:50.03951 2016-05-25T21:31:50.037Z 14577 TID-dnamc RepositoryImportWorker JID-ccc5ddfc8d4f361b66ccb4ce INFO: start
2016-05-25_21:31:52.78232 2016-05-25T21:31:52.781Z 14577 TID-dnamc RepositoryImportWorker JID-ccc5ddfc8d4f361b66ccb4ce INFO: done: 2.744 sec

        Same.

      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        @gajus thanks, looks like this was something different. I've put a fix already and should be merged for 8.8.3

      • Douwe Maan Milestone changed to %8.8

        Milestone changed to %8.8

      • Robert Speicher Status changed to closed by commit gitlab-org/gitlab-ce@1d2d67b9e35fe029250f6a11c2163637d0b186e1

        Status changed to closed by commit gitlab-org/gitlab-ce@1d2d67b9e35fe029250f6a11c2163637d0b186e1

      • username-removed-238606
        username-removed-238606 @gajus ·
        Author

        Any plans on releasing the 8.8.3? I don't understand how come you don't have hundreds of issue tickets for this – doesn't this affect everyone migrating from gitlab.com to github-ee?

      • James Lopez
        James Lopez @jameslopez ·
        Maintainer

        @gajus this is indeed scheduled for 8.8.3 - https://gitlab.com/gitlab-org/gitlab-ce/issues/17883

      • Robert Speicher Mentioned in commit pfjason/gitlab-ce@ac98845e

        Mentioned in commit pfjason/gitlab-ce@ac98845e

      • Robert Speicher Mentioned in commit pfjason/gitlab-ce@fa90dcc1

        Mentioned in commit pfjason/gitlab-ce@fa90dcc1

      Please register or sign in to reply