Import from github fails

Summary

See: https://gitlab.com/gitlab-com/support-forum/issues/1441#note_22133931 The Import fails importing a big repository with undefined method '+' for nil:NilClass @ahmadsherif

Steps to reproduce

Try to import github.com/xbmc/xbmc (which you might need rights for)

Expected behavior

The import should end successfully

Actual behavior

It get's stuck due to the undefined method '+' for nil:NilClass

Relevant logs and/or screenshots

@ahmadsherif Might be able to provide those

Output of checks

This bug happens on gitlab.com

Checking the project status, it only got around 2k PRs imported and no issues, so it failed during PRs. Actual importing wouldn't such error (a String instead of a JSON) since the code is wrapped in begin ... rescue block, so the failure has to do with code outside the wrapped code. I think only two places that use + has the potential to fail: in client.rb and in importer.rb, the latter is more likely to be culprit.

Cache keys used for GH importing has a TTL of 2 weeks (the code specifies 1.day but apparently it gets ignored), but I tried to check the keys of this project and found none, although the import started (and failed) a week ago, I don't know the reason behind that. However, it doesn't quite explain why the page variable would be nil since any call to increment_page has to be preceded with by one call to current_page which would make sure there's a current_page exists.

Maybe we shouldn't depend on Redis to store importing progress and use the database instead, with a job that cleans afterwards every now and then.

added 480950 gitlab-foss13300 labels

mentioned in issue gitlab-foss#28145 (closed)

mentioned in issue gitlab-foss#28183 (closed)

I tired a new import some days ago, doesn't seem to have improved.

mentioned in issue gitlab-foss#8938 (moved)

changed milestone to %9.3 in gitlab-foss

/cc @jameslopez

This will probably get fixed after this GitHub refactor is ported to the importer (at the moment it's only available for the rake task) and this follow-up is done.

added Deliverable in GitLab FOSS label

@jameslopez Is that something you / @dbalexandre can look into in the not too far off future?

changed milestone to %9.4 in gitlab-foss

@DouweM I can port the refactor made on the rake task to the importer for %9.4 in gitlab-foss

@dbalexandre Please :)

assigned to @dbalexandre

@dbalexandre What's the status here?

@DouweM I started working on this, but I'll don't have time to finish it today. Can we move it to %9.5 in gitlab-foss ? I'll be able to push an MR at the beginning of the next week.

changed milestone to %9.5 in gitlab-foss

mentioned in merge request gitlab-foss!12886 (merged)

closed via merge request gitlab-foss!12886 (merged)

Still not working, it fails even faster now. It doesn't even get the repository and no issues PRs are imported.

Edit: Tried another time, same result.

@Razzeee Could you add me as a collaborator on this project on GitHub? This way I can try to figure out what's going on. My GitHub handle is @dbalexandre.

reopened

I will talk to the others and see if we can do that. I'll report back.

@dbalexandre you should have gotten an invite now, please let me know if it works for you

@Razzeee Thanks! I got the invite. I'll report back.

moved from gitlab-foss#27429 (moved)