Pass environment variables when triggering a pipeline with $CI_JOB_TOKEN

Description

When triggering a new pipeline using $CI_JOB_TOKEN (used by Multi-Project Pipelines), it is not possible to pass environment variables in the same way they can be passed when using Personal Access Tokens. This is due to a different implementation of the trigger on the backend side, that lacks the ability to store variables that are passed in the call.

Proposal

Support passing variables when triggering a pipeline using $CI_JOB_TOKEN, in the same way we can pass variables to regular triggers.

Use cases

In order to deploy documentation with pages, I'd like to have the documentation building jobs and deployment in a dedicated project, not containing the source code or the the documentation itself. Pushes to the source-code project then trigger the pipeline in this doc-building project, but the latter would need to know exactly which branch/commit it is supposed to build the documentation for.

Example

Variables can be passed in the following way:

curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline

added ~155960 feature proposal labels

/cc @bikebilly

@bikebilly This is very important issue, I wonder if we can schedule it.

@bikebilly This issue is very important for our QA, can we schedule this?

added OKR label

@bikebilly @ayufan We somehow forgot about scheduling this, and I think it's a critical part of the MVC for multi-project pipelines. Any chance of us getting it into 9.5? It's also necessary for our OKRs.

added Stretch label

changed milestone to %9.5

added Deliverable and removed Stretch labels

assigned to @dosuken123

After discussing with @bikebilly we decided to make this Deliverable for %9.5 instead of https://gitlab.com/gitlab-org/gitlab-ce/issues/33281 which become Stretch.

cc @dosuken123

@ayufan @bikebilly We need to cut a scope for the %9.5 iteration.

I suppose at least we want to achieve passing Trigger variables.

This will look like

# gitlab-ci.yml
build:
  script:
    - curl --request POST --form "variables[PRE_CI_PIPELINE_SOURCE]=$CI_PIPELINE_SOURCE" --form "token=$CI_JOB_TOKEN" --form ref=master http://192.168.10.3:3001/api/v4/projects/13/trigger/pipeline

User can know the status by exposing pre-stage variables.

Here are examples for the usecases.

Command	Result
--form "variables[PRE_CI_PIPELINE_SOURCE]=$CI_PIPELINE_SOURCE"	You'll know how the pipeline was triggered at the previous stage with $PRE_CI_PIPELINE_SOURCE
--form "variables[PRE_CI_COMMIT_REF_NAME]=$CI_COMMIT_REF_NAME"	You'll know which branch was triggered at the previous stage with $PRE_CI_COMMIT_REF_NAME
--form "variables[PRE_CI_COMMIT_SHA]=$CI_COMMIT_SHA"	You'll know which commit-sha was triggered at the previous stage with $PRE_CI_COMMIT_SHA
--form "variables[PRE_CI_JOB_NAME]=$CI_JOB_NAME"	You'll know which job was triggered at the previous stage with $PRE_CI_JOB_NAME

@ayufan Any thoughts about https://gitlab.com/gitlab-org/gitlab-ee/issues/2772#note_35126760?

@bikebilly For our current needs, this would be sufficient. We only need a way to pass custom variables at this stage.

@dosuken123

I believe that giving you an information about the dependent pipeline is an orthogonal issue that we should track separately. We can achieve that quite easily, even today.

This issue is about allowing passing explicit variables, similar to Trigger variables.

@dosuken123 Could you create a separate issue to add additional predefined variables when the dependent pipeline is run?

mentioned in issue #2960

added EE Premium label