Skip to content

Add support for `OffPeak` configuration for autoscale mode

@ayufan proposed to add the OffPeak configuration for autoscale mode. It's something that we've discussed few months ago but haven't described it yet.

Let's consider such configuration:

[[runners]]
  name = "auto-scaled"
  limit = 100
  url = "https://gitlab.com/ci"
  token = "XXX"
  executor = "docker+machine"
  [runners.docker]
    image = "ruby:2.1"
  [runners.machine]
    IdleCount = 100
    IdleTime = 3600
    MaxBuilds = 1
    MachineDriver = "digitalocean"
    MachineName = "auto-scale-%s"
    MachineOptions = [
      (...)
    ]

Looking on [runners.machine] section we can see that we will have 100 machines in Idle state, waiting to execute new builds. When using machines created on any of public cloud providers it can be expensive to have at least 100 machines running all the time. And most of organizations have some sort of off-peak times. For example - looking at builds for GitLab projects we can see, that there are lot of builds from Friday to Monday (a work week when most of GitLab employees/contractors and also most of community contributors are working), and a very few builds at Saturday and Sunday (weekend!).

Why to pay for 100 machines running for two days (so 100 * 24 * 2 * cost of one instance), when - let say - only 30 of them will be used once for not more than an hour (so 30 * cost of one instance)?

To resolve this we could add a configuration options like this:

  [runners.machine]
    IdleCount = 100
    IdleTime = 3600
    PeakHours = "8-18"
    PeakDaysOfWeek = "Mon,Fri"
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0
    (...)

How would this work?

  1. Days from Monday to Friday (PekaoDaysOfWeek), from 8 to 18 (PeakoHours) are marked as Peak time.
  2. If we are in the Peak time, machines scheduler is using the IdleCount and IdleTime settings, so it keeps at a moment a 100 of machines in Idle state (of course using also limit and concurency settings as described in autoscale documentation). Each machine - if not used - is removed and replaced with a new one after 3600 seconds of beeing in IdleState.
  3. If we enter the Peak time, machines scheduler is switching to OffPeakIdleCount and OffPeakIdleTime settings, so it removes all Idle machines. From now on a machine is created only when it's needed.
  4. If we go out of the Peak time, machines scheduler is switching back to IdleCount and IdleTime, and so on.