Not to clean up between jobs

I have a job in YAML script that consists of compiling, testing and uploading the results. Part of this job which is the testing takes a long time. I tried to separate that task into three sub tasks for compiling, running the tests and uploading results.

In such setup I could separate tests into 3 parallel parts and run the whole build faster.

However I recognised that both the build directory and the gradle cache directory are cleaned up between jobs. It basically means I have to recompile on each job. The overhead of recompiling each time is so high that the overall run time will only increase.

Is it possible to disable this behaviour? like dedicating a directory to each commit and share it between jobs?

Assuming you're using docker runners, you need to use volumes to keep directories between builds. Everything else is destroyed; that's part of the point.

I tried this approach and it will not work. the problem is that if I mount a volume on the build path, all the builds will share the same context. I can not change the code checkout directory as the check out happens before anything from YAML file is executed. And even if I overcome all the above mentioned problems, I think removing build and temp directories is kind of hardcoded in runner, as I can see these line on very beginning of each build log after docker images are pulled successfully:

Fetching changes...
Removing .gradle/
Removing build/

I even tried to use env variables in build path in runner's TOML file but they get all escaped and final result is always the same directory.

In 8.2 we have a new "caching" feature: http://doc.gitlab.com/ce/ci/yaml/README.html#cache

However, this requires a runner 0.7.0 which will be released this week.

Milestone changed to v0.7.0

Reassigned to @ayufan

Status changed to closed

great, thanks for the info

running gitlab 8.2.1 and runner 0.7.2, I have no success using cache for this purpose.

maybe there is a misunderstanding about what I meant here. I want one "job" in a build process to do compilation and the next "job" to use compiled files for running tests and so on.

following is the part of my build script that should do this:

types:
  - assemble
  - test
  - preserve

cache:
  paths:
  - build/
  - .gradle/


AssembleCode:
  type: assemble
  tags:
    - jdk-8
    - nexus
    - gradle

  script:
    # run build
    - gradle assemble -P version=$CI_BUILD_REF

    # verify configurations
    - gradle setup --tests *.SetupTasks.validateConfigFiles



RunTests:
  type: test
  tags:
    - jdk-8
    - nexus
    - gradle
    - orientdb-service

  services:
    - mohamnag/orientdb:2.1.6

  script:
    # wait for DB
    - ping mohamnag__orientdb -c 1
#    - while ! nc -q 1 mohamnag__orientdb 2424 </dev/null; do sleep 10; done

    # rebuild DB
    - gradle setup --tests *.SetupTasks.rebuildDatabase

    # run tests
    - gradle cleanTest test -P version=$CI_BUILD_REF




PreserveArtifacts:
  type: preserve
  tags:
    - gradle
    - jdk-8
    - nexus

  script:
    # preserve build artifacts
    - rm -rf $BUILD_ART_DIR/
    - mkdir -p $BUILD_ART_DIR/javadoc/
    - mkdir -p $BUILD_ART_DIR/tests/
    - mkdir -p $BUILD_ART_DIR/resources/

    # build module's zip
    - gradle modZip
    - mv build/libs/*.zip $BUILD_ART_DIR/
    - mv build/resources/* $BUILD_ART_DIR/resources/
    - mv build/docs/javadoc/* $BUILD_ART_DIR/javadoc/
    - mv build/reports/tests/* $BUILD_ART_DIR/tests/

however both tasks RunTests and PreserveArtifacts does not seem to use the build/ and .gradle/ directories and try to assemble code.

Status changed to reopened

and I still see following lines on beginning of each job's log:

Removing .gradle/
Removing build/

@mohamnag You will see it, this is generated by git clean, but caching restores it later.

I had this problem too but I think it stems from a fundamental misunderstanding of the cache feature. What I (and, I think @mohamnag) want it to do is cache files between jobs of the same build. But what it does is cache files between builds of the same job.

In this example, AssembleCode, RunTests and PreserveArtifacts each have their own completely independent caches which are restored before each run. Whats more each branch has its own set of caches meaning if you tend to only push to each branch once when creating a merge request, your cache will almost always be empty.

@tamlyn yes, what I meant (from the first ticket) was caching between jobs of the same build.

I used to use Bamboo, and it had a pretty similar feature, so a job consisting of multiple steps could share artifacts between steps. The new cache feature however does not seem to do the this.

I have currently solved my problem by mounting an extra volume and copying shared artifacts between jobs to that one, but it could have been more transparent.

I agree with @mohamnag as well. If dependencies are generated in one step these dependencies are not available in the next step. It would be beneficial if they were.

@mohamnag @tamlyn @snumb130

Let's don't mix the caching with passing artifacts between stages.

Caching is not designed to pass artifacts between stages. You should never expect that you have the cache present. I will make the cache more configurable making it possible to cache between any jobs or any branch or anyway you want :) Cache is for runtime dependencies needed to compile the project: vendor/?

Artifacts are designed to upload some compiled/generated bits of the build. We will soon introduce the option (on by default) that artifacts will be restored in builds for next stages.

Closing this in favor of #336 which is better suited to handle that case.

Status changed to closed

mentioned in issue #2037 (closed)

I'm facing an issue where creating/extracting the cache takes up longer than the build time itself due to the size of the generated files. Having the option to tell the ci not to clean up between jobs instead of caching would definitely save me a lot of time! I'm talking 20+ mins wasted on creating the cache, then extracting it in the next job.