Runners running out of free space
We're running gitlab-runner-docker-cleanup on our runners but they still seem to run out of space. Logs:
time="2016-07-26T20:22:49Z" level=info msg="Freed bytes: 0 B files: 18446744073709550762"
time="2016-07-26T20:23:19Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:23:19Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:23:19Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:23:19Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:23:20Z" level=warning msg="Freeing disk space. The disk space is below the lower bound: 0 B trying to free up to: 20 GB"
time="2016-07-26T20:23:20Z" level=warning msg="Freeing files count. The free file count is below the lower bound: 1218848 trying to free up to: 2097152"
time="2016-07-26T20:23:20Z" level=warning msg="Failed to free disk space: no images or caches to delete"
time="2016-07-26T20:23:20Z" level=info msg="Freed bytes: 0 B files: 0"
time="2016-07-26T20:23:50Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:23:50Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:23:50Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:23:50Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:23:50Z" level=warning msg="Freeing disk space. The disk space is below the lower bound: 0 B trying to free up to: 20 GB"
time="2016-07-26T20:23:50Z" level=warning msg="Freeing files count. The free file count is below the lower bound: 1222899 trying to free up to: 2097152"
time="2016-07-26T20:23:50Z" level=warning msg="Failed to free disk space: no images or caches to delete"
time="2016-07-26T20:23:50Z" level=info msg="Freed bytes: 0 B files: 18446744073709550103"
time="2016-07-26T20:24:21Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:24:21Z" level=warning msg="Failed to inspect container 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39 No such container: 414ec80b151361843e00109b8085a7d507e12324eee579893f8cd62a0548bb39"
time="2016-07-26T20:24:21Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:24:21Z" level=warning msg="Failed to inspect container aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341 No such container: aab8c1f29d9929c6d3d76ca3c979136f769e642dcd7c03f4cf1ea4029ea00341"
time="2016-07-26T20:24:21Z" level=warning msg="Freeing disk space. The disk space is below the lower bound: 0 B trying to free up to: 20 GB"
time="2016-07-26T20:24:21Z" level=warning msg="Freeing files count. The free file count is below the lower bound: 1205094 trying to free up to: 2097152"
time="2016-07-26T20:24:22Z" level=warning msg="Failed to free disk space: no images or caches to delete"
time="2016-07-26T20:24:22Z" level=info msg="Freed bytes: 0 B files: 18446744073709550733"
The only thing it does seem to manage to delete is the image used for the jobs on the runner causing it to frequently have to be re-downloaded.