Cache not always restored on concurrent > 1 and docker executor

I don't know if it's an expected behavior but i have noticed that the cache is split for each docker concurrent container.

I'm experiencing a storm of failed jobs because the jobs are executed in different concurrent container and obviously the cache is not present.

I'm going to explain myself with an example:

.gitlab-ci.yml

cache:
  key: "$CI_PROJECT_ID/$CI_BUILD_REF_NAME"
  paths:
    - node_modules/

# Build job
build:
  type: build
  script:
    - npm prune
    - npm install

# Test job
test:
  type: test
  script:
    - npm test

config.toml

concurrent = 1

[[runners]]
  name = "node_wheezy"
  limit = 0
  disable_verbose = false
  tls-ca-file = ""
  executor = "docker"
  [runners.docker]
    image = "node:wheezy"
    privileged = false
    volumes = ["/cache"]
    cache_dir = "/cache"
    allowed_images = ["node:*-wheezy"]
    allowed_services = ["redis:latest", "mongo:latest"]

If i keep concurrent to 1 the cache is always restored but if i put 2 it's possible that build job is executed on concurrent-0 and test on concurrent-1 causing a fail.

Title changed from Docker cache not always restored on concurrent > 1 to Cache not always restored on concurrent > 1 and docker executor

The concurrent job gets it's own cache. This will most likely change in one of the upcoming releases where the cache will be shared.

Milestone changed to v1.2

Because this issue, I have to only set concurrent=1. I wish to fix that fatal-bug as soon as possible. Fighting!

Me too, this is very annoying, the only other solution is to repeat each step in every job ( npm install, ecc.. )

I have added the configuration for s3 caching, thought it might help, but it doesn't seem to be used. Bucket is empty.

@mcfedr Seems the same issue : #1198 (closed)

Milestone changed to v1.3

Added cache label

Milestone changed to %Backlog

Back on this issue...

Adding S3 as volume is cool but i think is useless to completely solve this problem, if i want the cache is to be fast and fast means local, i don't want to push and pull from remote AWS servers GBs of data.

I understand that each concurrent runner clone has a different cache, and it's good maybe but why gitlab spawns a new concurrent runner for a job that is executed after the previous one instead of using the same runner clone?

So here i propose some solutions:

• By default jobs should try to use the same concurrent runner
• Add an option to force the same concurrent runner during a build
• Max concurrency param on the project config

Why i'm proposing this solutions?

• S3 is a no option for us (as stated before)
• if i set concurrency to 1 then ALL my projects will run only 1 build at time
• if i set limit to 1 on the runner then it will not be able to run concurrently for multiple projects using it
• artifacts are not good because i have to set expiration and i can't keep them for future builds.

Mentioned in issue #336

@ayufan 8 month ago, you were talking about an upcoming release with shared cache. Do you have any news about this feature?

@ayufan I'm having the exact same issue as described above. Here is my use case:

.gitlab-ci.yml

image: node:6.9

cache:
  key: "$CI_BUILD_REF_NAME"
  paths:
    - node_modules

stages:
  - install
  - compile

install_node_modules:
  stage: install
  script:
    - npm install

compile_code_stage: 
  stage: compile
  artifacts:
    paths:
      - dist/
  script:
    - npm run deploy:stage
    
compile_code_prod: 
  stage: compile
  artifacts:
    paths:
      - dist/
  script:
    - npm run deploy:prod

A run down of what happens:

• install stage
  • install_node_modules runs on concurrent-0 - creating a cache of the node_modules folder
• compile stage runs in parallel:
  • compile_code_stage runs on concurrent-0 - restores node_modules from cache
  • compile_code_prod runs on concurrent-1 - tried to restore node_modules from cache, but the cache is empty and the build fails.

I can't believe more people haven't discovered this issue, it's been 10 months since it's been raised. Don't get me wrong, I appreciate all the hard work the people at gitlab have put into this. I just hope this issue get's prioritized.

Thanks!

@bmv437 I have the same issue. Did you have a chance to resolve it?

@NikolayGalkin The only "solution" I found was to use a shared cache via S3 (or in my case, a Minio server which is compatible with the S3 api). I used Minio because it's faster than S3 for me, since it can be hosted on the same machine as the runner.

Here's the documentation: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#the-runnerscache-section

@bmv437 we installed Minio in our CI-line, however it was pretty slow (like minutes). We had a node_modules cache op ~75Mb (pretty large I guess).

Any other suggestions. Using artifacts for the node_modules also is not an option since it will only fill up our Gitlab instance (and probably also be slow).

https://gitlab.zendesk.com/agent/tickets/79388 (internal use only)

added SP1 customer labels