Remove public/ directory from being cached

We use the same cache for all branches and master can potentially download the cache from other branches. Removing the public/ dir from being cached will prevent this. Also it doesn't make any sense to cache it as middleman doesn't support incremental builds :/

Probably fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/31278