Import/include common CI/CD content from one project into another's `.gitlab-ci.yml` (EE only?)

Description

When companies have many projects, they often have standardized testing and deploying processes, which are often reflected in .gitlab-ci.yml. We should make it easy to share these processes and DRY up CI/CD configuration by letting one project import/include from another project's repo; either its .gitlab-ci.yml or another specified file. By allowing multiple imports, configuration could be componentized with a mix-and-max approach to constructing .gitlab-ci.yml content.

Could be an EE feature, targeted at larger organizations.

Proposal

Add keyword include which contains a string or array of strings.

Use a fully-defined pipeline from another project:

include: same-group/another-project/.gitlab-ci.yml

Use template definitions:

include: same-group/template-repo/.gitlab-ci.yml

test1:
  <<: *predefined_template

Mix-and-match template definitions:

include:
  - same-group/template-repo/.gitlab-ci-rspec.yml
  - same-group/template-repo/.gitlab-ci-rubocop.yml

test1:
  <<: *rspec_template

test2:
  <<: *rubocop_template

Include from arbitrary file, URL, or specific branch:

include:
  - group/project/directory/file.ext
  - user/project/directory/file.ext
  - https://server/url
  - group/project#branch

Using predefined templates seems like the recommended best practice, but the first way is a great way to get started, or use if a company has a rigid reusable process. It's comparable to including all of Twitter bootstrap vs including only the components you use.

Out of scope

Mix-and-match raw (direct import into job):

test1:
  include: same-group/template-repo/.gitlab-ci-rspec.yml

test2:
  include: same-group/template-repo/.gitlab-ci-rubocop.yml

This last option might seem easier because you can ignore YAML templates, which are pretty awkward. But you quickly end up including the same file multiple times, buried deep in the config file, and that feels like an anti-pattern. Dropping this scope may also ensure that each included YAML file can be fully validated on its own.

Links

• Requested here: https://forum.gitlab.com/t/ci-common-configuration/3862
• Older proposal: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1258

How would you know which branch / tag of the file to include? It's tricky since in some cases you might want settings in the master to automatically change global ci settings, and in other cases you might want projects to have to opt-in to a global ci change update.

I'd be fine only supporting inclusion of master/head; at least to start. But I could also see including any arbitrary URL. The trick there is permissions.

FYI, I originally submitted the idea on the forums.

@gaff Use the default branch for the project, maybe? Admittedly that's not always optimal.

@markpundsack Agreed. It's better than nothing.

Another alternative would be to allow including a YAML file from the filesystem instead of directly from a url. That way we could checkout whatever branch we want (or even a tag). I have to do quite a lot of this (checking out other projects) in our builds anyway. In fact, part of our build clones project that just contains bash scripts that, among other functions, determines the best branch to use if a dependent project doesn't have the same branch name.

I would envision this to work like: git clone -b develop <my_config_repo> ../my-config-project-dir and in the .gitlab-ci.yml: !include ../my-config-project-dir/build-defs/build-config-1.yml

Hope that makes sense.

Indeed - you could even support config included via a url served up from gitlab itself?

In many ways this seems a duplicate of this issue: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1258

@gaff I didn't read that whole issue, but it does seem very similar.

I'm kinda against including local filesystem, mostly because it's outside of version control, but also because it can't be parsed until a job is running, which limits the types of things you could include. e.g. you could never specify only in a local file because that is evaluated on the main GitLab process before the runner sees the job.

But it brings up another interesting topic which is shared scripts that really simplify .gitlab-ci.yml. We've tended to have things like prepare_ci.sh checked into a project's repo and I really wish those things could be shared somehow. Git submodules come to mind, but I still wish there was a better way to compartmentalize CI configuration separately from a project's code. Plugins are the current hope there.

Yep, that issue is basically the same request. Should probably move that to this repo though, since it's not just a runner issue.

Perhaps an alternative solution is (for users who want it) to have your .gitlab-ci.yml hosted in a different project altogether? So in your project settings you simply choose a separate project (and branch / tag) for your config file rather than pull it from the local repo?

This could perhaps be combined with including other snippets from the local fs / local repo, to pickup project specific settings?

I strongly disagree with breaking YAML to support this. In my opinion, there is no reason that the YAML spec needs to be broken to provide this feature. The suggestions in gitlab-org/gitlab-ci-multi-runner#1258 all maintain YAML parsability meaning that it can have its core syntax validated by local tools, editors with code colouring won't break and IDEs that do yaml parsing also won't break (unless you use a map-merge with an alias from your import - see my next comment on gitlab-org/gitlab-ci-multi-runner#1258).

Excellent point @flungo! includes: could work just as well. I don't have strong opinions on the syntax, just on the functionality.

Mentioned in issue gitlab-foss#22559 (moved)

Mentioned in issue gitlab-foss#22771 (closed)

Is someone working on this issue (code wise)?

I am thinking of:

includes:
  - group/repository/directory/file.ext
  - user/repository/directory/file.ext
  - https://server/url

This would be extremely powerful for implementing centrally controlled build plans

Mentioned in issue gitlab-foss#21644 (closed)

Milestone changed to %Backlog in gitlab-foss

Added ~123454 ~131895 labels

The trick there is permissions.

This is greatly simplified now that pipeline runs have permissions based on the user that triggered them. This means reading another repo's configuration is reasonable as long as permissions are managed well on the projects.

@sspreitzer No, no one is working on it. I just marked it up-for-grabs if someone (you?) wants to pick it up.

And yeah, that includes syntax looks good. We can start by implementing it only at the top level and see how far that gets us. Good use of YAML templates should make the "Mix-and-match raw" version unnecessary.

Updated description, but went with include since it works for both singular and multiple inclusions.

mentioned in issue gitlab-foss#15041 (moved)

mentioned in issue gitlab-foss#24574 (closed)

mentioned in issue gitlab-foss#18157 (moved)

removed ~123454 ~131895 labels

added ~698938 122770 gitlab-foss724481 labels

mentioned in issue gitlab-foss#28592 (moved)

added ~123454 label

mentioned in issue gitlab-foss#29042 (closed)

mentioned in merge request gitlab-foss!5682 (closed)

mentioned in issue gitlab-foss#29412 (closed)

mentioned in issue gitlab-foss#32638 (moved)

mentioned in issue gitlab-foss#32829 (closed)

If the community wants to contribute this, we'll consider it for CE. Otherwise there's some discussion to make it EE only as it's not totally necessary, but a convenience.

changed milestone to %Next 3-6 months in gitlab-foss

added Accepting Merge Requests in GitLab FOSS label

I'm potentially interested in taking a stab at this. I'll talk a look at it over the next few days and post again if I think I can commit to it.

@alexives Did you manage to take a crack at this yet?

I need to look into it a little more still, I've been focusing on another merge request I have open. Fetching the files (at least by url) should be relatively easy, and getting the files from a branch/repo shouldn't be too bad.

The tricky part here is actually merging templates in the yaml files. Some of that is a feature of yaml. It's much easier to source the files and do a hash merge, but that precludes using the built in yaml templating. I'm going to dig around and see how other people solve this. Maybe RailsConfig handles this? Not sure if they actually merge files or if they do hash merge.

From an implementation standpoint, it looks like this will require changes in BOTH gitlab and in the gitlab-ci-multi-runner. (Someone please correct me if I'm wrong.) I'm going to do some more digging into that project and see if I can come up with an approach.

I wonder if we build something similar to ksonnet to make it possible? https://github.com/ksonnet/ksonnet-lib

@alexives I was hoping it would only require changes to gitlab rails, but I haven't looked into details.

includes:
  arbitrary_g_name:
    path: group/repository/directory/file.ext
    tag: latest
  arbitrary_u_name:
    path: user/repository/directory/file.ext
    # this would hit master
  arbitrary_site_name:
    url: https://server/url/raw

@alexives I've been working on an external tool to implement this feature (as mentioned in https://gitlab.com/gitlab-org/gitlab-ce/issues/29042#note_31050552), and have run into some of the same issues that I'm guessing you've seen. There's an aspect to this that I think needs considered - especially in the context of the enterprise use perhaps: the ability to review the generated CI pipeline before submitting it. The tool I've been working on assembles the .gitlab-ci.yml file pre-commit and allows for such a review process before being dispatched to the CI workers.

I think there are definitely some issues which can be hit with a tool that runs completely out of band, namely: that people need to remember to run it. I also think there are some benefits to being able to review the result of a tool which will control the process of building and potentially be deploying the code.

@VertigoRay Any reason for the arbitrary names? This would work just as well:

includes:
  - path: group/repository/directory/file.ext
    tag: latest
  - # this would hit master
    path: user/repository/directory/file.ext
  - url: https://server/url/raw

Additionally, using a list preserves order which means that conflicts can be resolved by having precedence based on ordering of the includes.

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1258#note_14072380 contains further advanced features and implementation thoughts that I had.

marked this issue as related to gitlab-foss#32829 (closed)

Please make this a CE feature

Do you thnk it would be possible to have a big .gitlab-ci.yml and only include some jobs?

We can use the &/* system to call differents jobs.

@jeremy.collin.lnk Do you mean you want to be able to have one large shared file with all the jobs in it and then be able to pick and chose which jobs you want to be included? My comment on https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1258#note_14072380 proposes some options which would allow you to filter the jobs that are included.

Unfortunately, if I am not mistaken the & and * syntax is handled by the YAML parser and so if this is implemented based on parsed YAML (doesn't make sense to tinker with the YAML parser in ruby), then references between files using that syntax would not be possible. Would be really nice I agree, but the filter and merge options I have in my proposal would hopefully allow you to achieve the same things.

---

@alexives Have you made any progress with this?

I definitely don't have time right now, but if this is still unimplemented in a couple of months, I'll see if I can make some time to have a look at this and possibly help with getting this as a CE feature.

My general thoughts about the process at the moment (having not dug into the code for this yet):

• Parse the .gitlab-ci.yml in the project.
• Inspect the includes, check permissions and fetch the required files for processing
• Do these two steps recursively on a depth-first basis (to allow includes to include other files) with permissions always being based on the project that is being built.
• As the recursion pops back perform a merge of the included file with its parent using the rules specified within the include in order to build an "effective .gitlab-ci.yml"
• Cache the "effective .gitlab-ci.yml" for each commit.
• When running a job, provide this "effective .gitlab-ci.yml" to the

With caching it would be nice to be able to view the "effective .gitlab-ci.yml" from the web UI and also provide an option to trigger a rebuild of this.

I like the idea of the mix-and-max approach. But in some cases, it would also make sense to remove or override details. We, for example, have tonnes of projects and now we evaluated a software to analyze code. It would be pretty cool to add a new stage. So that code analysis stage will be placed in the parent and any child will automatically get the new stage as well.

• But some of the projects don't need this stage because the analyzer tool is not for the language.
• Some projects simply don't need to be analyzed.

A feature like overriding, known from programming languages, would solve the problem.

Overriding:

• The result would be: ONE stage has passed. Recognized by the developer with the default naming - but different implementation.

Removing:

• The result would be: NO stage for analyzing has passed. Or will be marked as removed.

Our security team would like to have this feature to provide code scanning (for vulnerabilities, plain text passwords, etc) for the CI pipeline. It would additional need to include either force injecting a job or being able to run a report to get which repositories do not include this code scanning job.

The problem Jendrik points out about the tool not being for the language in the repository seems complicated as well. Could there perhaps be a way to add jobs to the pipeline from within a job?

It could perhaps behave like this then:

• scan for general bad practices - plain text password in code
• add jobs for identified scenarios - if Ruby, add Ruby code scanner to pipeline

mentioned in issue gitlab-foss#38366

I would like to offer my use case for consideration, for which I'd also need some kind of include statement.

Currently, we are maintaining a separate repository of build scripts (not .gitlab-ci.yml snippets!) and include it in our projects as git submodules at .gitlab-ci/. So our main scripts look a bit like this:

build:
  tags:
    - docker
  stage: build
  script:
    - source .gitlab-ci/docker-login
    - source .gitlab-ci/docker-build

The advantage is that we can avoid cramping fully fledged bash scripts into YAML and also do a lot of additional logic which is difficult to replicate in every single project we have. For example, our docker build includes advanced Docker caching, like load/save and --cache-from and also failure handling. We are very averse to copy-and-pasting, which we already do a lot with Gitlab CI (YAML snippets would be nice to avoid it).

Our current struggle is a relatively long repository clone process in every build step. I would like to use GIT_STRATEGY: none for several build steps, however this would mean that my build scripts would need to migrate back to .gitlab-ci.yml, since there might be no .gitlab-ci/ to source from.

A proper include statement would generate an "effective .gitlab-ci.yml" as mentioned before, which would work perfectly even without a clone. Personally, I'd be fine with only including from the local repository since I can handle every other case with submodules or subtrees and still keep the pipeline simple enough. There is AFAIK no known workaround for building a .gitlab-ci.yml from snippets and scripts without a current repository clone.

@flungo Organization is all. Your example works just as well; if not better (ordering).

changed milestone to %10.3

moved from gitlab-foss#20868 (moved)