investigate cleanup of obsolete DO droplets
Since we have an integration tests setup on our cookbooks (defined in template and gradually being adopted to rest of them), it'll be nice to ensure we clean up those instances. Currently, the DO integration does use the --destroy=always
flag and we do have a $USER
prefix to hostname to easily track'n'cleanup locally created instances, but it'd be nice to have a automated way to clean up them too.
Right now, there's three nodes that are left from (likely) failed pipelines and was not cleaned up:
- https://cloud.digitalocean.com/droplets/61948692
- https://cloud.digitalocean.com/droplets/62127530
- https://cloud.digitalocean.com/droplets/63008928
We use the cheapest 512MBram boxes for cookbook CI pipelines, so its not that big financially, its just nice to have (and prevents us from going over the limit on dev DO account).
Currently proposed solution is to automatically clean up the instances once a week on sunday, since all CI ones use the ci.gitlab.test
(and some just gitlab.test) domain in a reserved .test TLD. Meaning we can easily do:
- query DO for instances having
*.gitlab.test
fqdn - (optionally) filter out ones that are younger than few minutes to not interfere with (unlikely) sunday's pipelines
- destroy the rest
And that won't remove anything of importance in already volatile dev account.