Skip to content
Snippets Groups Projects
Commit f4e94c6f authored by Sean McGivern's avatar Sean McGivern
Browse files

Make usage ping scheduling more robust

On GitLab.com, we see that cron jobs sometimes don't run. This appears to be
because the process that polls for current cron jobs only has a 60 second
validity period, so if (for whatever reason) it misses those 60 seconds, we have
to wait until the next execution.

For the usage ping, this is particularly problematic, as that's only scheduled
to run once a week.

Changing it to run for every minute in a 10 minute period should work around
this, if the above diagnosis is correct. The job itself obtains an exclusive
lease for 24 hours, so rescheduling in quick succession is safe.
parent 5843a43c
No related branches found
No related tags found
No related merge requests found
---
title: Make usage ping scheduling more robust
merge_request:
author:
type: fixed
Loading
Loading
@@ -113,12 +113,14 @@ class Settings < Settingslogic
URI.parse(url_without_path).host
end
 
# Random cron time every Sunday to load balance usage pings
def cron_random_weekly_time
# Runs every minute in a random ten-minute period on Sundays, to balance the
# load on the server receiving these pings. The usage ping is safe to run
# multiple times because of a 24 hour exclusive lock.
def cron_for_usage_ping
hour = rand(24)
minute = rand(60)
minute = rand(6)
 
"#{minute} #{hour} * * 0"
"#{minute}0-#{minute}9 #{hour} * * 0"
end
end
end
Loading
Loading
@@ -398,7 +400,7 @@ Settings.cron_jobs['stuck_import_jobs_worker'] ||= Settingslogic.new({})
Settings.cron_jobs['stuck_import_jobs_worker']['cron'] ||= '15 * * * *'
Settings.cron_jobs['stuck_import_jobs_worker']['job_class'] = 'StuckImportJobsWorker'
Settings.cron_jobs['gitlab_usage_ping_worker'] ||= Settingslogic.new({})
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_random_weekly_time)
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_for_usage_ping)
Settings.cron_jobs['gitlab_usage_ping_worker']['job_class'] = 'GitlabUsagePingWorker'
 
Settings.cron_jobs['schedule_update_user_activity_worker'] ||= Settingslogic.new({})
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment