An error occurred while fetching the assigned iteration of the selected issue.
Do not update/delete: Banner broadcast message test data
Do not update/delete: Notification broadcast message test data
@aleix_maneko does https://about.gitlab.com/gitlab-com/ answer your question?
changed the description
There's also a blog post about this with further info:
added support label
@winh can you please let us know when the quota is reset? e.g. I started getting this message today for one of our projects, but it's June 1st so the current count should have been reset, no?
According to graph the job was run: https://performance.gitlab.net/dashboard/db/sidekiq-workers?orgId=1&var-worker=ClearSharedRunnersMinutesWorker%23perform&var-database=Production&from=now-24h&to=now. But it seems that we hit the SQL timeout on reset.
I did reset quota manually. Description of the problem is on the go.
The problem is with this query: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/clear_shared_runners_minutes_worker.rb#L10.
We have:
> ProjectStatistics.count
=> 2385330
This leads to SQL query timeout. Thus we do not reset quotas. And process even that retried do fail.
The simplest fix is to change the offending code:
ProjectStatistics.where('shared_runners_seconds > 0').
update_all(shared_runners_seconds: 0,
shared_runners_seconds_last_reset: Time.now)
NamespaceStatistics.where('shared_runners_seconds > 0').
update_all(shared_runners_seconds: 0,
shared_runners_seconds_last_reset: Time.now)
But this, we limit data set to a reasonable amount and we do not hit the timeout. However, this is still not ideal, as we should reduce the number of updates at a single time to some amount, like 1k-10k.
This is also what I did exactly run to fix the problem now.
Since this is resolved and the actual fix has to be made, closing this one: https://gitlab.com/gitlab-org/gitlab-ce/issues/33213.
closed