Large number of UpdateAllMirrorsWorker jobs in the cronjob queue

Started around 21:55 UTC and steadily growing at about 10K jobs per hour:

https://performance.gitlab.net/dashboard/db/sidekiq-stats?panelId=3&fullscreen&orgId=1&from=1502315576100&to=1502322166333

/cc @stanhu

I know Sidekiq-cron runs this every minute, but I wonder why these jobs didn't run and finish quickly.

/cc: @tiagonbotelho, @DouweM

added mirror label

It looks like these are running, but perhaps there aren't enough workers to pick these jobs up?

https://log.gitlap.com/goto/da2efa66c2b2f89f8e37e15294909896

Just checked that we have all sidekiq services running with be knife ssh 'roles:gitlab-base-be-sidekiq' 'sudo gitlab-ctl status'. We do, and workers are not overloaded.

Hmm, I see we do we schedule UpdateAllMirrorsWorker each time someone attempts to import a project. That might be part of the issue.

A more likely explanation is that each time a repository mirror finishes, it kicks off another UpdateAllMirrorsWorker: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/repository_update_mirror_worker.rb#L37

Just a quick scan of the Redis value suggests that is happening:

irb(main):004:0> PULL_CAPACITY_KEY = 'MIRROR_PULL_CAPACITY'.freeze
=> "MIRROR_PULL_CAPACITY"
irb(main):005:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 1
irb(main):006:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 3
irb(main):007:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 5
irb(main):008:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 7
irb(main):009:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 8
irb(main):010:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 6
irb(main):011:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 4
irb(main):012:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 5
irb(main):013:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 4
irb(main):014:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 3
irb(main):015:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }
=> 3
irb(main):016:0> Gitlab::Redis.with { |redis| redis.scard(PULL_CAPACITY_KEY) }

The mirror when it finished it kicks schedules a new UpdateAllMirrorsWorker if a certain threshold has been reached. But the scheduler runs with a lease which means two of them cannot run at the same time.

@pcarranza @DouweM I would say it was not a bad idea to just have the crontab run every minute and drop the threshold because honestly I cannot think of a elegant solution that solves this.

If capacity is continuously below max capacity (as it is now), the threshold will always be reached, so a new one will get scheduled each time a mirror finishes, which is apparently about 10K/hour.

I think we have two options here:

1. We can remove this threshold triggering of UpdateAllMirrorsWorker altogether and reduce the crontab time to something smaller like 2s

2. Be more aggressive with the threshold.

Personally I am in favour of the option 1. If we can add more backpressure without ever risking having 10k/hour being scheduled it would be preferrable

@tiagonbotelho If we only schedule up to capacity mirrors once a minute, we won't get a lot done. We'd want to schedule more often than that if we have mirrors to schedule.

What about instead of checking only that available >= threshold, also checking that wants to run >= threshold so that we can fill it up completely? That way we won't schedule a new scheduler if there is nothing to put into the hole that opened up. If there are fewer than threshold mirrors that want to run, they just have to wait until the next blip.

I think we should also use a lease around scheduling the scheduler so that we never schedule one more often than once every 2 seconds.

@tiagonbotelho With 1, we'll keep running schedulers every 2s even if there are no mirrors in the system at all, putting unnecessary strain on the DB and Redis.

@DouweM checking wants to run >= threshold and available >= threshold will also put a strain on the DB and Redis. Imagine all the mirrors doing that query when finishing up. What seems more expensive?

@pcarranza what are our current values for mirroring in the admin application_settings?

@tiagonbotelho That way it's at least limited by the number of mirrors in the system and how often those are run, which is less often than every 2s for small instances.

Yup, agreed I can start working on this asap and update this issues's description with this proposal.

mentioned in issue gitlab-com/support#754

added Platform label

changed milestone to %10.0

assigned to @tiagonbotelho

mentioned in issue #3136 (closed)

mentioned in merge request !2671 (merged)

closed via merge request !2671 (merged)

mentioned in commit da5fb49e

mentioned in commit 4afeff11