Increase parallelism of GeoRepositorySyncWorker

While testing gitlab-org/gitlab-ee!2130, I see that right now GeoRepositorySyncWorker fetches one repo at a time. With hundreds of thousands of projects, this is too slow and inefficient, especially if one repo is blocked for some reason.

We should figure out how to do this more efficiently. I see that @pcarranza has already put a lot of thought into how we should schedule project mirrors: https://gitlab.com/gitlab-org/gitlab-ce/issues/29218 I wonder if we can extract this logic out and reuse that.

/cc: @dbalexandre, @tiagonbotelho

Customer request for faster Geo sync (especially the initial sync): https://gitlab.zendesk.com/agent/tickets/78986

added customer label

I do not know if we could do it like we do in UpdateAllMirrorsWorker which async schedules a RepositoryUpdateMirrorWorker which takes care of the sync. @dbalexandre WDYT?

@stanhu @tiagonbotelho @dbalexandre if by dirty project we mean that we have a way of detecting when a project got updated on the primary side (by a push or a merge, for ex) then we are talking about the first issue we solved where we were syncing mirrors after a push with a backoff @tiagonbotelho. That is indeed a much simpler use case than the pull mechanism.

@stanhu do we have a cursor of some kind on the client side?

Regarding triggering a lot of syncs at the same time concurrently, that could easily lead to having the system thrashing, so it should be done carefully and in a controlled manner.

@pcarranza Yes, on the primary we log an event when a push happens. The secondary has a cursor that goes through the event table and marks projects as dirty.

@stanhu great, then we can use a much simpler algorithm -> https://gitlab.com/gitlab-org/gitlab-ce/issues/29258

@pcarranza Awesome, thanks. Now I remember reading the code to do this.

assigned to @dbalexandre

changed milestone to %9.4

mentioned in merge request !2351 (merged)

closed via merge request !2351 (merged)

mentioned in commit e916e727