Add GKE PVC for faster caches
Right now, we first download a cache, unzip it, and than use it. At the end of that build, we zip the cache and upload it again.
However, this process can be improved by writing to disk instead of writing to a disk over the network. Thats why it would be good if we could mount a volume and use that. On Kubernetes this is possible, and I think the following process makes sense:
- Copy the right files to the location declared
- We need some access control here probably, what folders can be accessed, are all the caches on the same disk?
- Run the job
- Write the cache to the disk again
- Here is where the problems can start forming, especially with a lot of concurrent builds
- What would be hard to enforce, but should work, is have a ref counter per cache directory, and if a job starts copying the cache, increase it
- Once a new cache is created, start serving this cache
- Once the old cache's ref counter is 0, remove it
I'd like to think of a better way to handle this, but so far this is the best I got. @ayufan told me that we might want to consider cache read only jobs, but thats not something we have right now.