feat(docker): turn branch write caching off by default

It turns out that for many docker jobs, writing the branch write cache
takes many times longer than the docker build itself, so make it
optional.

docker.md

@@ -100,10 +100,32 @@ but these can be overridden depending on project requirements.
 There are three caching modes:
-1. **For branches**:  `docker buildx` will read from the branch cache and the default branch cache, and write the branch cache.
+1. **For branches**:  `docker buildx` will read from the branch cache and the default branch cache.
+   For performance reasons, branches don't write to the cache by default,
+   but this can optionally be turned on.
+   See [Branch Write Caching](#branch-write-caching) for more information.
 1. **For main branch**:  `docker buildx` will read from and write to the default branch cache.
 1. **For tags**: `docker buildx` will read from the default branch cache and not write to the cache.
+#### Branch Write Caching
+
+Writing cached images to the registry can be very slow:
+often slower than a rebuild if the main cache  is relatively warm.
+
+For that reason, writing docker builds to the branch cache is disabled by default.
+
+It can be enabled using the `~docker-write-branch-cache` label on your MR,
+or adding by the `DOCKER_WRITE_BRANCH_CACHE="1"` variable.
+
+If you're finding that Docker builds on a branch are slow and could benefit from writing the cache,
+for maximum efficiency, turn branch write caching on temporarily,
+run a successful pipeline,
+and then turn branch write caching off again.
+This will allow the branch cache to be populated,
+while avoiding the costly cache write at the end of each docker build job.
+
+Note that Docker builds will attempt to read the branch cache even when branch cache writing is disabled.
+
 ### Chainguard
 If your project uses [Chainguard](https://console.chainguard.dev/overview) images,

docker.yml

@@ -115,8 +115,22 @@ include:
         docker_cache_args="
           --cache-from type=registry,ref=${CACHE_REGISTRY}:${BRANCH_REGISTRY_CACHE_KEY_SLUGIFIED}
           --cache-from type=registry,ref=${CACHE_REGISTRY}:${DEFAULT_BRANCH_REGISTRY_CACHE_KEY_SLUGIFIED}
-          --cache-to type=registry,ref=${CACHE_REGISTRY}:${BRANCH_REGISTRY_CACHE_KEY_SLUGIFIED},force-compression=true,mode=max
         "
+        if [[ $CI_MERGE_REQUEST_LABELS =~ /docker-write-branch-cache/ ]] || [[ $DOCKER_WRITE_BRANCH_CACHE == "1" ]]; then
+          docker_cache_args="
+            $docker_cache_args
+            --cache-to type=registry,ref=${CACHE_REGISTRY}:${BRANCH_REGISTRY_CACHE_KEY_SLUGIFIED}
+          "
+        else
+          echo "----------------------------------------------------------------------------------------------------"
+          echo "⚠️⚠️⚠️ WARNING ⚠️⚠️⚠️"
+          echo "Writing cached images to the registry can be very slow: often slower than a rebuild if the main cache"
+          echo "is relatively warm."
+          echo "For best performance review:"
+          echo "https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/blob/main/docker.md#branch-write-caching"
+          echo "----------------------------------------------------------------------------------------------------"
+        fi
+
       fi
       set -x