Run jobs only/except when there are modifications on a given path

It would be nice if we could run jobs if there are modifications on paths matching a regex. For instance, if a commit only modifies files in the test subdirectory, only test stage would be run. This is achievable in Jenkins using "included/excluded regions" in Version Control configuration. In the gitlab-ci.yml, we could have something like:

  only:
    - paths:
        - "^test/.*"

For completeness, this should also be supported on except.

Added ~19173 feature proposal labels

@antoine.gaillard13 Thanks for suggestion, we will consider it! /cc @markpundsack

Sounds like a great idea. One complication: the example given says "if a commit only modifies files in the test directory", but that's slightly different than running tests "if files in the test directory are changed". The latter is easy to do using the syntax above, but I don't think the former is.

Other common use cases would be to automatically ignore changes only to documentation or other files that don't affect the build/tests.

Milestone changed to %Backlog

Changed title: [gitlab ci] Run stages/jobs only when there are modifications on a given path → Run jobs only when there are modifications on a given path

@markpundsack You're right my example is wrong. It should be something like

build:
  stage: build
  only:
    - paths:
      - "^src/.*"
test:
  stage: test
  only:
    - paths:
      - "^src/.*"
      - "^test/.*"

@antoine.gaillard13

I think that it may be a little tricky when we consider the incremental changes.

If we have a merge request that has 3 different commits, where:

1. modify tests,
2. modify documentation,
3. modify CHANGELOG,

And we have three seperate pushes for each of the commits and we are processing changes incrementally. How the flow of creating builds should look like:

1. Should we compare changes to previous push?
2. Should we compare against branch that we are merging to?

intuitively it seems that it should take into account the branch that we are merging to.

Yeah, I think comparing to the target branch, at the point of the branch, makes sense. Basically, whatever changes are included in a merge request.

Of course the real problem is that we currently run tests against branches, not merge requests, so we don't know what the target branch is.

mentioned in issue #19555 (moved)

@ayufan Yes you're right. I'd also like to point out that we have to handle artifacts. If we have something like:

build:
  stage: build
  only:
    - paths:
      - "^src/.*"
  artifacts:
    - paths:
       - "bin/"
test:
 stage: test
 only:
   - paths:
     - "^src/.*"
     - "^test/.*"

and we run black box tests in the test stage against binaries built during the first stage. If we only modify test subdirectory, we should download artifacts from the last parent commit that generated them.

Having only: paths seems like a nice feature. So

same as https://gitlab.com/gitlab-org/gitlab-ce/issues/19813

mentioned in issue #19813 (closed)

mentioned in issue #18157 (moved)

+1 This would make it possible for me to independently build/test/deploy the subprojects that live in my master repository.

+1

Is possible to store a hash of the content of the file/folder and compare later if it change?

@markpundsack this came up in slack today: https://gitlab.slack.com/archives/C027N716H/p1496775158700189 Wanted to get it back on your Radar <3.

Thanks @lbot. It's definitely still on our radar, but it's pretty complicated and not going to happen soon. But maybe we can open it up for Accepting Merge Requests.

Also, now that I'm rereading old comments, I wonder if we do want to test for content of each push, not compare vs target branch. The latter is especially hard because we don't know the target, so for now, the former is all we could do anyway. But also, I think it might be the right answer regardless. If I push a bunch of commits, and one more push contains just docs changes, there's no reason to re-run all builds and tests just because the branch contains other changes. Likewise, for master, there is no target branch, so paths would be useless unless they applied to individual commits in the push.

I do see the concern mentioned in https://gitlab.com/gitlab-org/gitlab-ce/issues/19232#note_13096248 though regarding artifacts. It might work to grab the latest artifact from the same ref, but I imagine that backfiring at some point. But that's probably semantically the same as what you're doing with Jenkins and persistent artifacts/caches.

added ~122770 test labels

changed the description

added Accepting Merge Requests label

changed title from Run jobs only when there are modifications on a given path to Run jobs only/except when there are modifications on a given path

I cannot read what is in that slack link, because it is behind a login - can that content be shared here?

mentioned in issue #18041 (moved)

To be honest I think this should have more priority. Companies are emerging towards more modular systems architecture with microservices and monorepos and more and more people are adopting GitLab CI for its CI/CD process. Using GitLab CI with a large monorepo today is really bad because you don't want to build everything whenever there are changes. You would want to trigger the pipeline/stage/job based on only/except some file path (glob pattern).

With #18041 (moved) you could also create a script to check changes.

added pipeline and removed test labels

changed milestone to %Next 3-6 months

This would be great for our cloudformation templates

mentioned in issue #23455 (moved)

I think this is an important feature to have the flexibility to filter the unnecessary CI jobs.

Eagerly waiting for this... :)

In the absence of Tower (which is expensive), we want to use this with Ansible repos so we can run jobs with different tags when certain files have changed.

Workaround to stay on GitLab-CI (as if there wasn't enough awesome without this):

use some scripting such as lolaus to 'skip' if the code has not changed

A GitLab-CI native solution would be much much better because then the job would not have to run at all and the declarative nature of the .gitlab-ci.yml file self documents better than the custom script.

I would really love having this feature.. so I could define a pipeline for each "build" - and only trigger the pipeline - on changes in a relevant folder.. would be super nice with something like:

triggers:
  - changes_below /folder1