limit parameter in runner config.toml is ignored

Summary

limit parameter in runner config.toml is ignored whilst using the following config file and .gitlab-ci.yaml file, I get multiple jobs running in parallel on the same runner in spite of defining the 'limit' parameter on my runner.

Steps to reproduce

Use the following config.toml and gitlab-ci.yaml and the stage 'test' will run multiple times in parallel.

$ cat ~/.gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "ios-gitlab-ci"
  url = "redacted"
  token = "redacted"
  executor = "shell"
  limit = 1
  [runners.cache]


$ cat .gitlab-ci.yml
stages:
  - nightly
  - test
  - beta

before_script:
  - source /usr/local/share/chruby/chruby.sh
  - chruby ruby 2.3.1
  - export PATH=$PATH:/usr/local/bin

deploy_nightly:
  stage: nightly
  only:
    - development
  script:
    - ./jenkins.sh
  tags:
    - ios

test_project:
  stage: test
  script:
    - fastlane test
  tags:
    - ios
  artifacts:
    paths:
        - ./test-output/*
    when: always

deploy_beta:
  stage: beta
  only:
    - /^release_.*$/
    - tags
  except:
    - branches
  script:
    - fastlane beta


variables:
    FASTLANE_XCODEBUILD_SETTINGS_TIMEOUT: "60"

Actual behavior

More than one job will run at once. Because this happens in the same git working copy, all parallel running builds will break each other resulting in me having to limit all jobs to a single git branch in order to have a function Gitlab CI

Expected behavior

Jobs should be listed as pending whilst only a single job per runner executes.

Relevant logs and/or screenshots

Environment description

The runner in question is a specific runner locked to a specific project. The runner is running in user mode on MacOS 10.12.5

Gitlab: GitLab Enterprise Edition 9.2.2-ee b004167

Used GitLab Runner version

gitlab-runner:

$ gitlab-runner --version
Version:      9.3.0
Git revision: 3df822b
Git branch:   9-3-stable
GO version:   go1.7.5
Built:        Thu, 22 Jun 2017 10:57:56 +0000
OS/Arch:      darwin/amd64

changed the description

added customer label

ZD ticket from an EE customer: https://gitlab.zendesk.com/agent/tickets/80359

Possibly related to https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1521

@zj can you take a look?

Do you maybe have a multiple runner instances running?

Can you look at each job and verify which runner ID they receive?

I'm pretty sure that there is only one runner in play here.

$ gitlab-runner list
Listing configured runners                          ConfigFile=/Users/redacted/.gitlab-runner/config.toml
ios-gitlab-ci                                       Executor=shell Token=redacted URL=https://redacted

I have confirmed that when two jobs are running at once, as per the screenshot, both running jobs are using runner # 23. When looking in Project-> Settings -> "CI/CD Pipelines" -> "Runners activated for this project" lists only one runner (23). Whilst there are 9 shared runners available, these are not in play for these jobs as indicated by the fact that when looking at both running jobs, they both say they're running on 23

Some other details that might be of note.

Running this to output the runners config does show the settings I'd expect:

$ gitlab-runner --debug run
Runtime platform                                    arch=amd64 os=darwin revision=3df822b version=9.3.0
Starting multi-runner from /Users/redacted/.gitlab-runner/config.toml ...  builds=0
Checking runtime mode                               GOOS=darwin uid=501
WARNING: Running in user-mode.
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

Configuration loaded                                builds=0
metricsserveraddress: ""
concurrent: 1
checkinterval: 0
loglevel: null
user: ""
runners:
- name: ios-gitlab-ci
  limit: 1
  outputlimit: 10000
  requestconcurrency: 0
  runnercredentials:
    url: https://redacted/ci
    token: redacted
    tlscafile: ""
    tlscertfile: ""
    tlskeyfile: ""
  runnersettings:
    executor: shell
    buildsdir: ""
    cachedir: ""
    environment: []
    preclonescript: ""
    prebuildscript: ""
    postbuildscript: ""
    shell: ""
    ssh: null
    docker: null
    parallels: null
    virtualbox: null
    cache:
      type: ""
      serveraddress: ""
      accesskey: ""
      secretkey: ""
      bucketname: ""
      bucketlocation: ""
      insecure: false
      path: ""
      shared: false
    machine: null
    kubernetes: null
sentrydsn: null
modtime: 2017-07-19T16:34:03-07:00
loaded: true
  builds=0
Waiting for stop signal                             builds=0
Metrics server disabled
Feeding runners to channel                          builds=0
Starting worker                                     builds=0 worker=0
Trying to load redacted.crt ...
Dialing: tcp redacted:443 ...
Checking for jobs... nothing                        runner=redacted

My suspicion is that something is awry because this is setup on MacOS and thus setup in user mode, not system, however my initial attempt to install the runner was in system mode. I did remove /etc/gitlab-runner also /Library/LaunchAgents & /Library/LaunchDaemons no longer contain the gitlab runner plists, they are correctly located in ~/Library/LaunchAgents. However, the following output makes me suspicious that something internal to gitlab-runner is looking in the wrong place for a runner config.toml file:

$ gitlab-runner list
Listing configured runners                          ConfigFile=/Users/redacted/.gitlab-runner/config.toml
ios-gitlab-ci                                       Executor=shell Token=redacted URL=https://redacted/ci

$ sudo gitlab-runner list
Password:
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml

N.B. that /etc/gitlab-runner/config.toml does not exist, it was deleted after successfully installing the runner in user mode. Restarting the system does not fix this issue.

@ayufan The customer at https://gitlab.zendesk.com/agent/tickets/80359 is asking for an update on this issue.

Can you check a list of processes that are running of gitlab runner or gitlab ci multi runner?

@ayufan There are 2 gitlab-runner processes running

Actually, there are usually two, but sometimes three.

It should always be one. Check how your system is configured and who spawns the other processes.