Skip to content
Snippets Groups Projects
New issue look: Off

Multiple pipelines

    • Closed (moved) Issue created by Mark Pundsack @markpundsack

    Description

    Note: This is not a priority, just capturing thoughts from several different issues into a single proposal.

    I read somewhere that all CI/CD systems are either: individual jobs, pipelines, or graphs. GitLab CI is clearly "pipelines" and trying to do graphs at all might just be wrong. But having said that:

    • People want to be able to have multiple pipelines that are completely independent, but described in separate files.
    Docs     ⟼⬤⟶⬤
Android  ⟼⬤⟶⬤⟶⬤
iOS      ⟼⬤⟶⬤⟶⬤

    or

    master            ⟼⬤⟶⬤⟶⬤
feature branches  ⟼⬤⟶⬤
    • People want to be able to have a CI pipeline that must pass before a separate CD pipeline can run, and want to describe them in separate files.
    CI ⟼⬤⟶⬤
           ➘
CD          ⬤⟶⬤⟶⬤
    • People want to be able to do complex CI/CD configurations that include trees, or parallel branches that should run independently, branched from different points.
    ⟼⬤⟶⬤⟶⬤
       |   ↳⬤⟶⬤
       ↳⬤⟶⬤⟶⬤

    Perhaps we can stop short of taking a generalized "graph" approach, and think instead of multiple "pipelines", with some dependencies between them (perhaps leveraging dependencies we need to support cross-project pipelines). By either forward or backward reference to one pipeline triggering another, and supporting multiple pipelines running concurrently, we effectively have created a backdoor for enabling this kind of complex behavior, while still keeping our "pipeline" core.

    In terms of implementation, it's possible to define these pipelines in a single .gitlab-ci.yml file, but as stated above, people want separate files. A high-leverage way to handle that would be to include separate pipeline definitions into the master .gitlab-ci.yml (https://gitlab.com/gitlab-org/gitlab-ce/issues/20868). But perhaps there's some justification for supporting them in completely separate files. One such reason is to have different .gitlab-ci.yml in sub-directories, which only apply to files changed in those sub-directories. Intuitively, I'd want logically-separate pipelines declared in separate files.

    Proposal

    • Allow declarations of multiple (named?) pipelines (optionally in separate files, via includes or pointers or ???)
    • Allow declaring regular conditions for triggering pipelines such as only and except
    • Allow declaring file-based conditions for triggering pipeline such as file or directory regex (pushes may trigger multiple pipelines with matching regexes)
    • Allow declaring pipeline-based dependencies to trigger pipeline (e.g. run pipeline X iff pipeline Y succeeds)
    • Allow declaring job-based dependencies to trigger pipeline (e.g. run pipeline X iff job A in pipeline Y succeeds)

    Note: There may be a natural parallel between pipeline/job dependencies and artifact dependencies, but since sometimes artifacts are managed outside of our artifact system (such as docker images, environment state, or runner state), these dependencies need to work with or without artifacts.

    Links / references

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply