Import forked repositories asynchronously to prevent large repositories from timing out

Preserve synchronous mode of adding a repository since some tasks (e.g. restore/check) require the former.

One side bug fix is that the ProjectCacheWorker is also run, updating the repository size and commit numbers properly. Previously all the values were set to 0 when a project were forked.

• Closes #2388 (closed)
• Closes #2400 (closed)

Added 1 commit:

• a98d6a2b - 1 commit from branch gitlab-org:master

Added 1 commit:

• 080762f3 - Import forked repositories asynchronously to prevent large repositories from timing out

mentioned in issue #2400 (closed)

@stanhu how would this look in the UI? What do I see as a user after forking a repository, before the git clone behind the scene has finished?

Added 1 commit:

• 1adb78ce - Import forked repositories asynchronously to prevent large repositories from timing out

@jacobvosmaer Users would see a blank repository for a big job with the message: "Project was successfully forked. Please wait a few moments for the repository to appear." I was going to look into putting in the spinner used for importing a repository from an external source, but I didn't want to conflate those changes with these.

Added 1 commit:

• 33183990 - Import forked repositories asynchronously to prevent large repositories from timing out

@stanhu sounds great

@dzaporozhets what do you think about this.

Forks fail on gitlab.com now for large repos because we do them in Unicorn. Any reason not to let Sidekiq do the forking?

@jacobvosmaer we should fork async with Sidekiq

Added 1 commit:

• 0a4f1e21 - Always execute project fork as an async task and leave project creation as-is

Added 57 commits:

• 0a4f1e21...6630a06c - 55 commits from branch gitlab-org:master
• a6067f1a - Import forked repositories asynchronously to prevent large repositories from timing out
• 4b5ac786 - Always execute project fork as an async task and leave project creation as-is

@dzaporozhets Thanks, that makes things simpler. I was trying to figure out how to make the UI show the status of the forked repository job. This is identical to what the import process does. Does anyone have a problem with using import_url and import_status to store the state of the forked repo import?

mentioned in issue #2400 (closed)

Added 13 commits:

• d6c07a90...99bbbb5c - 12 commits from branch gitlab-org:master
• 9a0132cc - Import forked repositories asynchronously to prevent large repositories from timing out

@jacobvosmaer Can you check/recreate the satellite for this repo? It is claiming that there are merge conflicts when there are none.

Added 1 commit:

• c2e10eaa - Import forked repositories asynchronously to prevent large repositories from timing out

It looks like another push made the problem go away. Odd.

Added 1 commit:

• 4f544ede - Log errors in forking

Sorry @stanhu must have been the gitlab.com gremlins ( @jnijhof and me) deleting/recreating satellites this week.

Added 69 commits:

• 4f544ede...a5bb85f8 - 66 commits from branch gitlab-org:master
• ebc5afda - Import forked repositories asynchronously to prevent large repositories from timing out
• 9aca70d5 - Log errors in forking
• 0ebfbf0a - Use import_status to track async import status and give feedback to the user

Added 1 commit:

• 95417941 - Fix test

I'm pleased that I've found a way to reuse the existing import structure to display "Import in progress" message to the user.

The only issue I'm running into is an artifact of my local dev environment with GDK on ports 3000 and 4000:

Uncaught SecurityError: Failed to execute 'replaceState' on 'History': A history state object with URL 'http://localhost:4000/test/vgstation' cannot be created in a document with origin 'http://localhost:3000'.

Somehow the X-XHR-Redirected-To field is showing port 4000 even though I access http://localhost:3000, which is causing the JavaScript to deny refreshing back to the main project page. I don't think this is an issue in a normal production system.

reflectRedirectedUrl = function() {
    var location, preservedHash;
    if (location = xhr.getResponseHeader('X-XHR-Redirected-To')) {
      location = new ComponentUrl(location);
      preservedHash = location.hasNoHash() ? document.location.hash : '';
      return window.history.replaceState(window.history.state, '', location.href + preservedHash);
    }
  };

Since this is a significant change to move async forks, please test this out and see if you run into other issues!

Added 1 commit:

• f9aae6fa - Fix test now that flash notice is not guaranteed

Added 1 commit:

• ada442fa - Import forked repositories asynchronously to prevent large repositories from timing out

Ok, the issue I was having was solved via gitlab-development-kit!69 (merged).

Reassigned to @dzaporozhets

Added 1 commit:

• 9995f080 - Import forked repositories asynchronously to prevent large repositories from timing out

Milestone changed to 8.0

FYI, the one weird thing is that the "Importing in Progress" message just says git clone --bare since there is no import_url. We may want to just omit that part.

@stanhu ok lets make an issue for this once it get merged

@stanhu nice re-use of import code for forking! ✨

Status changed to merged

Yay!

mentioned in commit c0c314c6

mentioned in issue gitlab-com/support-forum#325 (closed)

Thanks @stanhu

mentioned in issue #2491 (closed)

mentioned in merge request !1295 (merged)

mentioned in commit 4d2b67ac

mentioned in commit dad9de66