Pass CI build artifacts between stages

Provide option to pass artifacts from previously run builds (from previous stages) to current running one.

For compiled software, this is a big hole in the current CI solution. Right now, we can't have separate build and test stages, because when the test goes to run, the compiled binaries are gone, and would have to be built again. I really look forward to this feature being implemented.

I'm assuming that this feature would cause all files collected via the artifacts key in .gitlab-ci.yml (and only those) to "magically" re-appear in later stages? That would be excellent.

@jonathon-reinhart This is planned for 8.3.

@ayufan Still on track for 8.3?

Milestone changed to 8.4

Reassigned to @ayufan

mentioned in issue #4263 (closed)

Do we have some more information available on the design of this? I'm curious how it would work getting those artifacts from one stage to the next if there are multiple jobs for the first stage.

For example, let's say we have some python application, since it's not compiled we don't need a build stage, but we do have a test and a deploy stage. In the test stage there are two jobs, one using Python 2 and one using Python 3, simply because we want to test our code using both versions of python. The deploy stage should take one of those versions, let's say our production is still using Python 2, and package it into a docker container or so. How would we configure so that the deploy stage uses the build artifacts from the Python 2 test job?

mentioned in merge request !2437 (merged)

mentioned in commit c19795df

8.4 is released but I don't think this feature is shipped.

mentioned in commit tnir/gitlab-ce@c19795df

Milestone changed to 8.6

Does this need closed because gitlab-org/gitlab-ci-multi-runner!80 is merged? Or is there more to do, e.g. documentation?

We miss the .gitlab-ci.yml configure to tune it.

Hoping this not move to next milestone again

@xiemeilong-ok One way to ensure that doesn't happen would be to submit a merge request which adds the missing features.

mentioned in issue #13559 (closed)

mentioned in issue #14211 (closed)

mentioned in merge request !3182 (merged)

mentioned in commit ca3fc229

Milestone changed to 8.7

is there a way to disable artifact passing? i want each task keep it's own.

@hjanuschka Define job with empty dependencies:

job:
  dependencies: []

@axil Could we add this to documentation?

Status changed to closed

We did release 1.1 with support for all features.

We did release 1.1 with support for all features.

Nice job @ayufan! I look forward to trying this out!

thx man!

@ayufan Just to clarify: is artifact-passing enabled by default, unless you add dependencies: []?

If so, this seems like a bad, dangerous idea to me, as it defeats the isolation between stages that CI is intended to provide.

I think you should be required to call out dependencies from other jobs whose artifacts you require.

Update: I forgot that artifacts are only passed from preceding stages to subsequent stages. In that case, it probably isn't very dangerous. It's still a little concerning that simply updating to the newest versions of GitLab + runner will cause builds to change behavior.

@jonathon-reinhart I do understand your concern. Not sure if there's a good way to introduce such change, since it was supposed to be there from the beginning. I assumed that this is expected behaviour, of what people will think how it is supposed to work.

Is there something that we can improve with this feature?

Mentioned in commit pfjason/gitlab-ce@c19795df

Mentioned in commit pfjason/gitlab-ce@ca3fc229

Could we add this to documentation?

Added https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7575

If artifact-passing is enabled by default (without dependencies set), which previous job does the next job get artifacts if there are multiple jobs producing artifacts on the previous stage?

@anatolyborodin Yes, enable by default. From all previous jobs in all previous stages.