Make distributed cache to be shared between runners

I believe you can set the cache key value, isnit ?

The key directive allows you to define the affinity of caching between jobs, allowing to have a single cache for all jobs, cache per-job, cache per-branch or any other way you deem proper.

This allows you to fine tune caching, allowing you to cache data between different jobs or even different branches.

@ebuildy link to your text? i.e where your copied text originates? what key are you talking about?

you said "it's because cache key is identified by runner-id" so I am speaking about the key of the cache ^^

Documentation is here ==> https://docs.gitlab.com/ce/ci/yaml/README.html#cache

@brodock you are speaking about gitlab-ci config, which in my example is "ci-tools", but the problem is in s3 path, which in my example is "162eb344"

so no, problem still remains

@glensc setting up the cache sometimes can be confusing.

We have the cache from the perspective of the developers (who consumes a cache within the build) and the cache from the perspective of the runner.

Depending on which type of runner you are using, it can act differently. For example, in gitlab.com we use autoscaling runner, which can use an external S3 service to store your cache.

For the autoscaler runner, the cache key you define in gitlab-ci.yml is unique for the project. So during a single pipeline run (of one or multiple jobs), whoever uses the same cache key will get access to the same data, but they can't "share" in the same run data with each other (like a shared / networked folder).

See more details for the autoscaler configuration: https://docs.gitlab.com/runner/install/autoscaling.html

Do you know what type of runner you are using? You can also check the compatiblity chart to understand the implications and limitations of each type: https://docs.gitlab.com/runner/executors/#compatibility-chart

@brodock i'm using static docker runner with S3 configured as cache backend. (all on promise installations: gitlab-ce, gitlab-ci-multi-runner, minio). i can scale it manually with docker-compose scale runners=3, which would just create 2 more docker containers on the same node.

i've already identified why the cache is not shared between runners and proposed solution. it's described in issue initial description.

if i do not use S3 cache backend i do understand that each docker runner runs it's own container and cache is just the docker volume. however S3 makes it possible to share cache with the runners as i've already outlined in issue initial description.

my problem is not in .gitlab-ci.yml level, but runner level that runners can't share cache. actually in my installation they do share cache via the hack i described.

# docker-compose ps
             Name                   Command       State   Ports 
---------------------------------------------------------------
gitlabcirunner_gitlab-runner_1   /gitlab-runner   Up            
gitlabcirunner_gitlab-runner_2   /gitlab-runner   Up            
gitlabcirunner_gitlab-runner_3   /gitlab-runner   Up            
gitlabcirunner_gitlab-runner_4   /gitlab-runner   Up            
gitlabcirunner_gitlab-runner_5   /gitlab-runner   Up            

# docker-compose exec --index=1 gitlab-runner env |grep TOK
CI_SERVER_TOKEN=162eb344....
REGISTRATION_TOKEN=zPTqB9...

# docker-compose exec --index=2 gitlab-runner env |grep TOK
CI_SERVER_TOKEN=162eb344...
REGISTRATION_TOKEN=zPTqB9...

the all runners (ab)use the same CI_SERVER_TOKEN to be able to share cache, and they all 5 appear as just one runner in gitlab admin (162eb344).

as for the projects sharing cache, it is already unique in the url, in the example i gave: project/133. so that part is okay, projects can't steal each-one's cache.

@brodock can you give your example of cache url when using docker+machine?

@glensc I guess you are right. @ayufan please correct me if I'm wrong but the S3 cache was designed mostly with the autoscaler in mind. Trying to use it with isolated, manually defined runners, probably will shard the caches with the runner-id. It makes sense to be able to share cache with manually bootstraped runners overriding the path somehow.

Added feature proposal label

@brodock can you show your cache path used by docker+machine setup for comparison? i think you need to enable runner DEBUG option to see used cache urls.

@glensc I'm sorry, I don't have one running I can access and debug.

@brodock: i found this meanwhile: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1226 and https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1232#note_4832116 by @ayufan

he used the same hack as me to solve the cache sharing problem: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1226#note_11888355

based on those comments, the current solution is for "most cases", so should be okay to accept MR for "least cases".

will be submitting PR soon with S3_CACHE_ID variable proposed in this issue, defaulting to runner token to preserve current behaviour.

Added ~1009246 label

changed title from s3 cache not shared between runners to Make distributed cache to be shared between runners

Extend CacheSettings with:

Path to allow specify of custom storage path,
Shared to drop runner/<runner> from the generated path, by default set to false.

To generate a path:

- if shared:
  
    http://<s3-server>/<path>/project/<id>/<cache-key>

- if not shared:
  
    http://<s3-server>/<path>/runner/<runner-token>/project/<id>/<cache-key>

mentioned in merge request !417 (closed)

mentioned in merge request !423 (merged)

closed via merge request !423 (merged)

mentioned in issue #336

Make distributed cache to be shared between runners

Description

Proposal

Designs

Child items ...

Activity

Admin message

Admin message

Make distributed cache to be shared between runners

Description

Proposal

Activity