/var/lib/docker/volumes does not get cleaned up
On our boxes running gitlab-ci-multi-runner we see lots of orphaned (?) volumes in /var/lib/docker/volumes, which fill up the disk to a point where jobs can no longer start.
Being a grumpy system administrator, I put the following in a crontab:
# CAUTION
# Delete docker volumes created more than 2 hours ago
find /var/lib/docker/volumes/ -maxdepth 1 -mmin +120 -exec rm -rf {} \;
This is a bad idea if you expect docker-encapsulated CI builds on your server to last for longer than 2 hours, or if you run long-running docker containers that have nothing to do with CI on the same server.