Increased activity of mirroring leading to higher GitLab.com load

As of 14:00 UTC today we're seeing an increase in mirroring activity:

A similar pattern is shown for the other mirroring queues/jobs. The pattern also repeats itself every hour, which is less than ideal. As a result the project_mirror queue is currently throttled at 80% capacity to ensure other jobs are not impacted too much.

@brodock @tiagonbotelho since both of you are involved with mirroring, could you try to look into this to see what change in 8.17 (which we deployed today) might be causing this?

mentioned in issue #1696 (closed)

@tiagonbotelho Can you have a look here?

changed milestone to %9.0

@DouweM @yorickpeterse are we already using the global minimum that we got merged last week?

like I said to @yorickpeterse I think this is due to the fact that we merged the Global Minimum mirror sync time and the default value was 15 minutes since it makes sense to be that value for the instances running GitLab. So with this line of thought changing the global minimum should fix our issue Hope it helps!

Apart from the number of jobs in Sidekiq I'm thus far not really seeing a negative impact (e.g. on the DB or other jobs). However, given our size (and size to come) I'll move the sync interval to once per hour to give Sidekiq some more room to breathe.

closed