Runner locked to nothing after registering with a token not tied to a group or project

Summary

We power down GitLab and the Runner Docker containers outside of business hours. When a new Runner is launched each morning, it is registered automatically with GitLab using a registration token not tied to any particular project or group. The Runner's (surprising) new default behaviour of starting locked means that the runner will be locked to nothing and so it will never pick up a job.

Steps to reproduce

1. Launch GitLab with a pre-set token not tied to a project or group.
2. Use this token to register a fresh Runner.
3. Runner will start locked to nothing and will never pick up jobs.

Expected behavior

Runner starts not-locked or locked to all groups or fails to start with a fatal error.

Environment description

Shared Runners v10+ running against a private GitLab instance. Both applications being run in Docker containers

Used GitLab Runner version

10.0.0+

Thanks for the report @inhumantsar.

cc @bikebilly @markpundsack - I don't think I've seen this in my I2P testing on 10.0, but FYI.

INITIAL_RUNNER_TOKEN, or effectively https://docs.gitlab.com/ce/ci/runners/#registering-a-shared-runner seems to result in a condition where the runner is effectively useless.

@inhumantsar Could you confirm that the runner does show up under /admin/runners, as well as what the properties show for that runner? (active / tagged / locked / protected)

It does show up in /admin/runners with the shared and locked flags. Here's an example of what I saw using the 10.1 betas. I'm currently using 10.0.0 and the behaviour is identical.

[image: Inline image 1]

@inhumantsar Can you add REGISTER_LOCKED=false to the environment for the container that you spin up?

OK so... I guess I've been hit by a separate issue as well.

If I include --locked false in the command line params before --executor, the registration complains about an invalid executor. If I include it after --executor then it registers fine, but it's still showing up as locked. I've tried --locked with and without quotes around false (I've snipped some of the irrelevant options from this output)

bash-4.3# gitlab-runner register --non-interactive   --name "gitlab-runner-master"   --run-untagged   --docker-image "docker:latest"   --executor "docker+machine"   ...
Running in system-mode.                            
                                                   
Registering runner... succeeded                     runner=3vgXdhai
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!


bash-4.3# gitlab-runner register --non-interactive   --name "gitlab-runner-master" --locked "false"  --run-untagged   --docker-image "docker:latest"   --executor "docker+machine"   ...
Running in system-mode.                            
                                                   
Registering runner... succeeded                     runner=3vgXdhai
PANIC: Invalid executor specified                  
Unregistering runner from GitLab succeeded          runner=a3d2ae77


bash-4.3# gitlab-runner register --non-interactive   --name "gitlab-runner-master"  --run-untagged   --docker-image "docker:latest"   --executor "docker+machine"   ... --locked false
Running in system-mode.                            
                                                   
Registering runner... succeeded                     runner=3vgXdhai
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 

Using the ENV var instead of the command line parameter works, the runner registers and isn't immediately locked.

So in summary, the --locked flag is a problem in conjunction --non-interactive, but the environment variable works as designed? That sounds like a flaw in the implementation.

@WarheadsSE That, plus the fact that the shared token + locked means that nothing gets picked up.

@tmaczukin This sounds pretty significant if shared runners are just dead.

@joshlambert I've seen it in my testing today with I2P.

@inhumantsar You might prefer to use --locked=false. Thanks for the issue we will try to debug that and verify what is wrong.

@inhumantsar @WarheadsSE As @ayufan wrote, use the --locked=false option. It's a known weakness of used CLI library - it handles boolean flags as present-true/not present-false. If you want to directly pass the value you need to use the --flag=value pattern.

As for the main problem, we've confirmed that it's a wrong implementation in GitLab CE/EE that affects Shared Runners when locked is set to true.

In two services related to Runners and jobs we're checking if Runner can_pick? a job (for both Shared and Specific Runners):

• https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.0.0/app/services/ci/update_build_queue_service.rb#L5,
• https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.0.0/app/services/ci/update_build_queue_service.rb#L13,
• https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.0.0/app/services/ci/register_job_service.rb#L26.

Where can_pick? is defined as https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.0.0/app/models/ci/runner.rb#L114 which uses assignable_for?(build.project) which is defined as (https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.0.0/app/models/ci/runner.rb#L176):

def assignable_for?(project)
  !locked? || projects.exists?(id: project.id)
end

Since Shared Runners don't have projects assigned directly this indeed ends with jobs not starting there.

After a discussion with @ayufan we've agreed that this is bad implementation. locked flag was designed to prevent Runner's from being assigned to other projects and it should affect only UI and business logic related to moving/assigning Runners. It should not affect jobs scheduling.

We'll update the assignable_for?(project) method to:

def assignable_for?(project)
  is_shared? || projects.exists?(id: project.id)
end

which should fix the problem. We'll add the fix as fast as possible and release it with one of upcoming patch releases.

added Next Patch Release bug labels

I'm deploying the runner alpine-v10.0.2 via kubernetes, and new runners with a shared token still come up as locked. I've tried with both REGISTER_LOCKED to false and without it as well, still see the same behaviour.

@gdmello The fix was added in GitLab CE/EE 10.0.2, not in the Runner. You need to update GitLab :)

@tmaczukin - that makes sense. Thanks!