Adding variables to pipelines schedules

Resources

FE @MadLittleMods

Description

Add custom variables when you schedule a manual pipeline, they will be added to the existing one already defined at project level. If the variable is already there, it will be overwritten with the new value.

Proposal

on variable functionality:

• a user can edit existing variables
• by editing its text contents
• but can also, as a faster way, just delete a whole row of variables with the - button
• as soon as one has some content in it a new one appears

visual implementation:

Links / references

#30882 (closed)

Documentation blurb

When you run or schedule a pipeline manually, you may want to add or overwrite variables that are passed to jobs in order to modify the behavior just for that specific instance. In the same place you request the manual pipeline to be created, you can now also specify an additional set of variables that are added to the list of variables already defined for the project, overwriting existing ones in case they're already present.

mentioned in issue #30882 (closed)

Since there is backend work associated with this, and it hasn't been scheduled, I'm adding the backend label and unassigning myself for the time being.

removed assignee

Is there a way to tell a job that it is scheduled if you want to run a certain type of job on the schedule vs on commit without using variables? Like a stage build?

In one of our use cases we have a dev environment which gets updated on commit and then a stage env that gets built daily so it is somewhat stable for UAT testing.

+1

In my current setup I use variables sent with the trigger, but I'd love to switch from my own cron job to the pipeline schedules.

@bikebilly Yes. I started that work as part of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11682.

added Stretch label

removed Stretch label

added Deliverable and removed Stretch labels

changed the description

@ayufan Do you need any FE work on this or do you want the BE to do everything? I think it would be nice to use some jQuery to add new form elements. I can help you with that if you want. I am assigned to work on the FE here so let me know.

We have two parts, Backend who needs to prepare groundwork, and Frontend which needs to expose these settings. Yes, we may need to add some jQuery to make it nice.

@jschatz1 I know it's pretty small, but you might also consider using Vue for the variables list here. We're already using it for interval pattern selection (source) in this form, and where this form is headed (https://gitlab.com/gitlab-org/gitlab-ce/issues/30675), I think the more we can do in Vue, the better.

@brycepj thanks. @ayufan will you let me know when the BE is ready for the FE?

@brycepj interesting idea, it seems that we can mount Vue for variables only. Do we have to make API for that?

@ayufan I don't think so. As long as you expose existing variables in the view and accept new key/value pairs on form submission, you shouldn't need to create an API.

Oh. This is great then because it would follow the same design as we have for existing secret variables. This allows us to have Backend very quickly.

mentioned in merge request !12134 (merged)

added UX ready label

Will stay assigned for any additional UX

assigned to @dimitrieh

assigned to @dosuken123

@bikebilly There is a couple of things we need to discuss.

• Priority
  How do variables of pipelines schedule take precedence?
  Reference: https://docs.gitlab.com/ce/ci/variables/README.html#priority-of-variables

• Security
  Are all variables considered as public? Do we need to encrypt like Project Secure Variables?

/cc @ayufan

assigned to @MadLittleMods

changed the description

@bikebilly Also do we need a "Protected" option for scheduled pipeline? which project secret variables have.

@bikebilly I think it's better to explicit here. I assume you meant

1. Trigger variables (take precedence over all) or scheduled pipeline variables
2. Secret variables or protected secret variables
3. YAML-defined job-level variables
4. YAML-defined global variables
5. Deployment variables
6. Predefined variables (are the lowest in the chain)

This list is referred from the document

@bikebilly Ah, I just realized the "Protected" option is absolutely unnecessary. Because the target branch is always specified in Scheduled Pipeline.

Then, the rest of matters is whether we make "Scheduled pipeline variables" secret.

FYI, "Trigger variables" is not secret. https://docs.gitlab.com/ce/ci/triggers/README.html#making-use-of-trigger-variables

If your project is public, passing the token in plain text is probably not the wisest idea, so you might want to use a secret variable for that purpose.

As for the security of schedules:

• If you are owner of schedule (as developer) or master => you can read, modify and delete,
• If you are developer => you can just list, not read,

This allows only owners and masters to read variables assigned to the schedule. It prevents other developers from hijacking schedules, but allows master to fully control them. Master already has access to Secret Variables.

@ayufan Do we need to encrypt values by attr_encrypted as well as Ci::Variable does?

Yes. All variable values should be stored encrypted.

Specification Proposal for BE implementation

mentioned in issue #33980 (moved)

mentioned in merge request !12328 (closed)

mentioned in merge request !12341 (closed)

is there a different design for scheduled pipelines list that includes showing variables once set? I don't see one (and this could be the right choice, so security is guaranteed).

@fabio as far as i know not. There isn't to much screen real estate available and i think these are value that are good to be visible only in the deeper/inspectable view.. only accessible by the right people as you said for security reason. Aside from that.. there can be many variables, plus we have a description field.. where someone could state what is really necessary to understand this

I've just realized I don't need to add new endpoints for manipulating variables. Because the change will happen when user pushes "Add schedule" or "Edit schedule", not when user pushes plus/minus button (i.e. AJAX).

@MadLittleMods AM I correct? If you're expecting new endpoints, let me know.

@dosuken123 note that @ayufan and I discussed this toward the beginning of this issue. tldr: you're right, this shouldn't require new endpoints.

@dosuken123 No new endpoints needed. Just need to save those variables on form submission and have those variables available in the HAML to render when you edit an existing schedule.

@MadLittleMods @brycepj Thank you for the clarification. I'll finish the BE MR as soon as possible.

@MadLittleMods I was thinking about a better workflow between FE and BE. And I think it's better to use an intermediate branch instead of targeting master directly.

So it's going to be like

The benefits are

• We don't need to merge temporary codes into master (e.g. TODO: FE do somthing here).
• We can write feature specs before merging into master branch.

What do you think?

/cc @zj

@dosuken123 An intermediate MR works for me . In the past I have just targeted the FE MR against the BE branch, merge the BE branch, and re-target FE against master for final merge.

@MadLittleMods Yes, I've seen a lot of patterns of collaborating between FE and BE, and your suggestion also makes sense, especially small changes.

Well, anyway, let's try an intermediate MR! I'll create the empty MR later.

mentioned in merge request !12372 (merged)

@MadLittleMods Now you can change your MR's target branch to feature/intermediate/32568-adding-variables-to-pipelines-schedules. https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12372. Also please remove the Closes #32568 from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328.

@dosuken123 If this is not in the handbook, consider adding it, maybe a new page. It would be nice to have a proper procedure for this.

@zj Yes, if we can find a value through the experimental git-strategy, let's consider updating handbook. Anyway, is there any default workflow when FE and BE work together?

@dosuken123 Not to my knowledge. Sometimes 2 MRs, sometimes 1

@ayufan I disagree with the new security implications. One strategy to make transient failures surface is to divert from master at some point and every 4 hours or so run the schedule. Your current approach makes that hard for me, as developer, to achieve. Is this by design?

mentioned in issue #34328 (moved)

as decided in chat, on variable functionality:

• a user can edit existing variables
• by editing its text contents
• but can also, as a faster way, just delete a whole row of variables with the - button
• an empty row with a + button is always available
• as soon as the previous one has some content in it a new one appears

cc: @MadLittleMods @dosuken123

updated issue description

changed the description

changed the description

edited description with the following:

on variable functionality:

• a user can edit existing variables
• by editing its text contents
• but can also, as a faster way, just delete a whole row of variables with the - button
• as soon as one has some content in it a new one appears

visual implementation:

users can edit variables only if they are the owner of the schedule or in general?

@bikebilly if they are either master or owner :)

mentioned in commit aeb31904

@MadLittleMods @ayufan said that he hates https://gitlab.com/gitlab-org/gitlab-ce/issues/32568#note_33182488, so let's forget it and target master I'll create a concrete proposal and update Handbook someday.

@dosuken123 I'm saying that either we merge that separately (unlikely to happen, as in BE there are some TODO changes). I just propose to merge these two branches now into one and point it to master. Review them together now :)

mentioned in issue omnibus-gitlab#2542 (closed)

@ayufan Ah, yes. Technically, we're saying the same thing. The difference is whether we merge into intermidiate 3rd branch or either of two.

@dosuken123 Never mentioned hating it , only comment about it, https://gitlab.com/gitlab-org/gitlab-ce/issues/32568#note_33183522 ahh, I see you are saying @ayufan hates it.

I targeted the FE MR at the BE branch because the integration branch doesn't have the BE in yet. Once things get stable, we can merge BE into the integration branch, point FE at the integration branch and merge.

@MadLittleMods I closed an MR of integration branch and please never mind. Let's merge FE->BE or BE->FE, if we're ready. That's a usual workflow in GitLab I think.

Anyway, it's better to discuss https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12341#note_33627643 and https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12341#note_33653773, instead

mentioned in commit 2cd462ae

mentioned in commit cf5e3125

mentioned in issue #34518 (closed)

mentioned in commit d0bb7bf8

mentioned in commit 47970ea5

mentioned in commit c45a118b

mentioned in commit c51da64e

mentioned in commit 5576214d

mentioned in commit 888ce892

mentioned in commit cd2d435d

mentioned in commit 2dba0cf0

mentioned in issue #34805 (closed)

closed via merge request !12372 (merged)

mentioned in commit a77158ff

mentioned in issue #34741 (closed)

mentioned in merge request !14785