autoscaling tries to delete machines that don't exist
Summary
Running on GCE, the multi-runner tries repeatedly to delete machines that don't exists. The machines don't exists on GCE, nor in docker-machine.
Steps to reproduce
See the log below
Expected behavior
if there aren't any machines listed in docker-machine, the multi-runner shouldn't try to delete anything.
Relevant logs and/or screenshots
First: no machines listed in docker machine:
jobevers@gitlab-autoscale-runner:~$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
Lets stop and restart the multi-runner:
jobevers@gitlab-autoscale-runner:~$ sudo gitlab-ci-multi-runner stop
jobevers@gitlab-autoscale-runner:~$ journalctl -fu gitlab-runner
[...]
Sep 06 23:21:26 gitlab-autoscale-runner systemd[1]: Stopping GitLab Runner...
Sep 06 23:21:26 gitlab-autoscale-runner systemd[1]: Stopped GitLab Runner.
Upon starting, the log shows that we try to delete machines. This behavior continues indefinitely.
jobevers@gitlab-autoscale-runner:~$ sudo gitlab-ci-multi-runner start
jobevers@gitlab-autoscale-runner:~$ journalctl -fu gitlab-runner
-- Logs begin at Wed 2017-09-06 22:39:20 UTC. --
Sep 06 23:21:55 gitlab-autoscale-runner gitlab-runner[16973]: time="2017-09-06T23:21:55Z" level=warning msg="Removing machine" created=2.931725846s name=runner-4069eb59-auto-scale-runner-1504735649-e48c12b2 reason="machine is unavailable" used=34.34899ms
<nil>
Sep 06 23:21:55 gitlab-autoscale-runner gitlab-ci-multi-runner[16973]: time="2017-09-06T23:21:55Z" level=warning msg="Removing machine" created=2.931725846s name=runner-4069eb59-auto-scale-runner-1504735649-e48c12b2 reason="machine is unavailable" used=34.34899ms
Sep 06 23:21:55 gitlab-autoscale-runner gitlab-runner[16973]: time="2017-09-06T23:21:55Z" level=info msg="About to remove runner-4069eb59-auto-scale-runner-1504735649-e48c12b2" name=runner-4069eb59-auto-scale-runner-1504735649-e48c12b2 operation=remove
<nil>
Environment description
Running on GCE, using docker+machine.
jobevers@gitlab-autoscale-runner:~$ sudo docker info
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 2
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 8
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.10.0-32-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 586.1MiB
Name: gitlab-autoscale-runner
ID: K2PH:22AH:3IZF:TYWX:Y6YA:GIXM:HQDR:VMYV:BI3X:EODJ:25ES:RDWH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
Live Restore Enabled: false
WARNING: No swap limit support
jobevers@gitlab-autoscale-runner:~$ docker-machine --version
docker-machine version 0.12.2, build 9371605
Used GitLab Runner version
$ gitlab-runner --version
Version: 9.5.0
Git revision: 413da38
Git branch: 9-5-stable
GO version: go1.8.3
Built: Tue, 22 Aug 2017 13:35:13 +0000
OS/Arch: linux/amd64