Possibility to cache folders in build directory

I'd recommend mounting a volume as cache and adjust your .npmrc file to use the cache on the mounted volume.

ah sorry, i thought you're using a docker runner. You want to use a cache for a shell runner? then just use a directory outside your git repository...

@ayufan i don't have the permission to close issues, but this is not an issue for me. It could be a candidate for an faq section, or tag it with question here.

Let's leave it open, because this is nice issue and it should be resolved properly.

Milestone changed to v0.6.0

Added improvement label

Hi,

I tried the solution suggested by @widerin as follows (I'm using the docker executor):

Added a volume in config.toml called /npmcache
In my .gitlab-ci.yml, I added this:

before_script:
  -  npm config set cache /npmcache
  -  npm install -g grunt-cli'''
  -  npm install

When the runner is running I verified that that the /npmcache dir was being populated But for whatever reason, npm does not seem to use the cache for subsequent builds, and as a result there seems to be no speed up from making the npm cache persistent (the dependencies are downloaded and compiled on each build). No idea why this is the case.

So I tried caching the node_modules directory instead. Node doesn't allow changing the node_modules directory location easily, so I just added - ln -s /node_modules node_modules to my .yml after adding a /node_modules volume in config.toml The .yml looks likes this:

before_script:
  -  ln -s /node_modules node_modules
  -  npm install -g grunt-cli
  -  npm install

test:
  script:
    -  grunt ci_test --verbose

This reduced my build/test times from ~19 minutes to ~3 minutes

Can we create a data-container or a cached volume using the gitlab-runner:cache image which is shared over all build types and trashed after the whole .gitlab-ci.yml scripts finished?

What i want to achive is a shared folder to avoid running npm install again in every type. Since we basically support build pipelines it would be great if we could do something like this:

image: centos:7
types:
- build
- qa
- test
- deploy

build:
  type: build
  script:
    - npm install

quality-assurance:
  type: qa
  script:
    - npm run linters

test:
  type: test
  script:
    - npm run test-protractor-abc

deploy:
  type: deploy
  script: ...

mentioned in issue #157 (closed)

Also will apply to make this feature. Since for example if I have vagrant getting up at the beginning, then every time the folder .vagrant is deleted, since I have it in my .gitignore and because of that I need to create virtual machine from scratch, instead of using existing one.

+1 to this. npm install builds are painfully slow. Sequential builds should really use the same node_modules folder. And given that other products offer already available solutions, I believe this should be addressed duly.

I want to solve that with 8.2 release.

I really like the Travis CI syntax to cache directories:

Would be great to have a similar solution for Gitlab CI.

@ayufan Is this still scheduled for 8.2? We are considering upgrading our Gitlab instance before EOW, but we might wait for this to come up if it's going to see the light of day in the near future.

mentioned in issue #258

@nfantone Yes, it's.

Milestone changed to v0.7.0

Reassigned to @ayufan

Status changed to closed

This will be live in 8.2.

I upgraded to 8.2.

I use a gitlab-runner with shell execution and tried to add this to my build file:

cache:
    paths: 
    - node_modules/

Unfortunately, I get the following error:

gitlab-ci-multi-runner 0.6.2 (3227f0a)
WARNING: Defined cache is not supported for that executor
Using Shell executor...
Running on around-media-gitlab-ce...

I would like to keep the node_modules/ directory cached in between builds

I read here that I need to use a file outside my directory, but this node_modules/ gets generated by doing npm install. Anyone that can help me out?

@thijs1 but your runner is still gitlab-ci-multi-runner 0.6.2.

Aha @widerin, indeed, now I see it. I just saw it too in the changelog of gitlab-runner. :)

Thanks for the fast answer though!

I have upgraded gitlab-ci-multi-runner to 0.7.1 (ee7a001) and added:

cache:
    paths:
    - node_modules/

to my .gitlab-ci.yml file but I'm still not entirely sure how this works. I can see that builds now output something like:

Archiving cache...
INFO[0000] Creating archive cache.tgz ...               
INFO[0075] Done!

before finishing, but my node_modules directory is still being deleted before starting a job and dependencies are installed as normal (no sign of caching whatsover), even on subsequent builds. @ayufan, what am I missing here?

@nfantone Not sure. They will get deleted, but they should get restored from tar archive.

Hi, I waited for this feature, however that tar/untar works quite slow for projects with thousands of files in node_modules. So I went back to making symlinks to /cache folder.

@inwaar How the caching works is the subject to change. It's tied to the code of GitLab Runner. So we are free to improve it overtime :) The most important is that we have it working, and the GitLab returns paths that we can consume anyway we want later.

@inwaar The 0.7.1 had problem with archiving files multiple times. The 0.7.2 should improve the speed significantly.

Based on the discussion above, it is implied here that the same cache should be used for all of the build jobs. However, on my server with Gitlab 8.2.1 and runner 0.7.2, a separate cache tarball is created for each job.

My .gitlab-ci.yml looks like this:

stages:
  - setup
  - test
  - build
  - publish

artifacts:
  paths:
    - build/

cache:
  paths:
    - node_modules/
    - jspm_packages/

install_deps:
  script:
    - npm install
    - ./node_modules/.bin/jspm install
  stage: setup

lint:
  script:
    - ./node_modules/.bin/jshint src
  stage: test

build:
  script:
    - ./node_modules/.bin/jspm run bundle-sfx main-node.js    build/node.js        --format cjs
    - ./node_modules/.bin/jspm run bundle-sfx main-node.js    build/node.min.js    --format cjs --minify
    - ./node_modules/.bin/jspm run bundle-sfx main-browser.js build/browser.js     --format global --global-name window.MyCompany
    - ./node_modules/.bin/jspm run bundle-sfx main-browser.js build/browser.min.js --format global --global-name window.MyCompany --minify
  stage: build

push:
  script:
    - npm config set registry https://nexus.mycompany.com/content/repositories/npm
    - npm publish
    - npm config delete registry
  stage: publish
  only: tags

When the install_deps job runs, at the end I see:

Archiving cache...
INFO[0000] Creating archive cache.tgz ...               
INFO[0003] Done!

... and I can inspect the tarball that is created and see all of the files there.

However, when it gets to the next job, test, it cleans up (as expected), which includes:

Removing jspm_packages/
Removing node_modules/

... but it never unpacks the tarball, leading to the following error:

$ ./node_modules/.bin/jshint src
-su: line 25: ./node_modules/.bin/jshint: No such file or directory

Is this supported, or not? If yes, what's wrong in my case? If not, are there plans to make it work?

@kohenkatz Currently the cache is per-job. The 8.3 will introduce option to "group" cache and reuse it between different jobs. It's currently done this way, because you can test against different versions of NodeJS, and the cached objects may not be valid for different image.

I'm thinking about adding:

cache:
  name: "$CI_BUILD_REF"

You could use any name you want. You could group by branch, tag, job name, stage name, some variable. It will be flexible. However, the default will always be per-job caching.

@ayufan Thanks. Looking forward to seeing this.

@ayufan Is there any example on how the group option is used? Can there be per job and grouped cache at the same time? If yes how, if no is it planned?

Thank you

I'm getting

gitlab-ci-multi-runner 0.7.2 (998cf5d)
WARNING: cache is not supported by selected executor and shell
Using Shell executor...

Am I correct in assuming that cache is not supported on windows platforms?

@laitentier Unfortunatelly, yes :( We will have it done in upcoming 1.0 version that will be released around the GitLab 8.4 release.

@ayufan Seriously? Better get to it then, 8.4 is scheduled to be released 22nd this month ;) 8 days left!

Keep up the good work, you're doing an amazing job! Until this is fixed, I'm sure we can get around it. @inwaar how are you fixing your symlinks?

@laitentier Yes :)

@laitentier You will also have artifacts support for Windows with built-in in GitLab browser :)

Is it possible to cache folders outside the project path?

For example, unlike node, .NET DNX stores all downloaded packages in a global folder - /root/.dnx/packages. I'd like to be able to share the downloaded packages across jobs which belong to different stages.

When I create two jobs with a shared cache (through the new group name), sharing works fine, but only for paths relative to the project folder (for example ./my-local-path).

However, when I attempt to use the following cache definition:

cache:
  group: my_group
  paths:
    - /root/.dnx/packages/

the runner yields:

  Archiving cache...
  INFO[0000] No files to archive.

I checked - the folder is there and it is full of files, but the cache archiving seems to be skipping it for some reason.

On another note, what is the best way to debug the build script generated by gitlab-ci-multi-runner? Being able to peek behind actions such as Archiving cache... would be helpful.

Currently not. It may change in future. You may want to symlink local directory to /root/.dnx.

@ayufan - thanks for the prompt response!

How will symlinking work for cache restore action? Won't I have to create the sym-link before cache restore kicks in?

It doesn't matter. The symlink should be live before the command you want to execute. I mean the symlink other way around :))

Try this:

cache:
  paths:
  - dnx-cache/

before_script:
  - mkdir dnx-cache
  - ln $PWD/dnx-cache /root/.dns

Good stuff @ayufan!
This totally worked (with a few tweaks):

cache:
  group: my_group
  paths:
    - dnx-cache/

before_script:
  - mkdir -p /root/
  - mkdir -p dnx-cache
  - ln -s $PWD/dnx-cache /root/.dnx

Is there any way to speed up archiving/restoring of the cache (for example turn off compression)?
Also, is there a way to tell a job to restore the cache, but not archive at the end?

@vassilvk The archive at the end will happen only if any of the files did change.

Currently not. Maybe it would be good to disable compression by default.

Thanks @ayufan.

Yes, I think adding an option to enable/disable compression (and maybe make it disabled by default) would be a very useful update.

I am using cache as a way to speed up my builds, but in reality my build process slows down because of the overhead of the archive-and-restore operations.

I'm sorry to tell you but I'm experiencing the same. My builds are taking 14 seconds longer.

Besides, I'm having another problem: On Windows I'm experiencing difficulties archiving my node_modules directory. I enable git.long_paths (or something like that), but it's complaining about files it cannot locate.. And surprise surprise, it can't find the long paths ;-) And of course, it's having difficulties restoring the archive as well.

Why are you archiving? Why not using the linking as default? It appears to be much quicker. Perhaps you could work something out by using hard links instead of soft links. This way the data will exist on the disk.

i don't understand people claiming here about slower builds when using cache. if it is slower, don't use it :-) if you ever built an app with npm in your tests with a lot of dependencies you gonna really like this cache! i experienced - to mention something positive here - at least 500% faster builds than without cache.

@ayufan just an idea, but thinking about allowing different compression levels for zip or just plain tar, which should be fastest to be configureable in config.toml what do you think?

@widerin Yes. I think that I'll disable compression completely for cache. It doesn't make sense.

@widerin I see no point for introducing extra option for that.

You maybe interested in checking out the artifacts passing: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/80

Possibility to cache folders in build directory

Designs

Child items ...

Activity

Admin message

Admin message

Possibility to cache folders in build directory

Activity