Docker executor: Various goroutines race with Cleanup()
In various places in the Docker executor, the go s.removeContainer(...)
pattern is seen as part of error handling. In particular, failing calls in Prepare()
may start one or more of these cleanup goroutines.
The Cleanup()
method of the Docker executor runs a (nominal) close against its docker_helpers.Client instance, after waiting for all successfully-created containers to be removed. However, the goroutines created in Prepare()
may still be running when docker_helpers.Close(s.client)
is called.
I noticed this by getting a SIGSEGV out of a branch where I was calling s.client = nil after the Close() call.
In master, the impact is that the call to s.client.RemoveContainer
(from the cleanup goroutines) may come after the call to CloseIdleConnections()
- resulting in a new socket being created, and kept idle indefinitely.
Rather than starting these goroutines, we could add a failures
member to the executor and call removeContainer
in the Cleanup()
?