Tag schedules and switch jobs on or off according to tag

Description

Currently you can have multiple schedules, but each one must run the whole pipeline. From 9.3 you can switch on and off jobs according to whether or not the pipeline was called by a schedule, but still not per schedule. This cuts out a big use case of having multiple schedules.

Proposal

Give each schedule a 'tag' and extend the only/except filters to allow references to schedule tags.

Links / references

The following issue is related, but I don't think it's possible to switch on or off jobs according to the value of a variable.

https://gitlab.com/gitlab-org/gitlab-ce/issues/32568

Documentation blurb

Use cases

If you have an intrinsically non-deterministic testing system (which typically becomes harder to avoid, the higher level or the more applications integrated into your tests), you might want periodic re-runs of the same test over a period of development in order to gather statistics about reliability. You might do this as part of a drive to make your tests more deterministic and ultimately completely deterministic.

Meanwhile this requirement can be satisfied by periodic re-scheduling of your pipeline (if one job runs one instance of such a test). From 9.3, this is fine if you only have one test: you can use only/except to select which jobs your pipeline should run. If you have several tests, it is hard to make different schedules run different jobs in your pipeline.

Feature checklist

Make sure these are completed before closing the issue, with a link to the relevant commit.

• Feature assurance
• Documentation
• Added to features.yml

added ~19173 feature proposal labels

I, too looked at the pipeline schedule documentation and felt like something wasn't quite right:

job:on-schedule:
  only:
    - schedules
  script:
    - make world

job:
  except:
    - schedules
  script:
    - make build

To me, the entire concept of .gitlab-ci.yml specifying whether or not a job should be run during a scheduled pipeline feels mis-placed.

A pipeline schedule should be considered as a user clicking the "run pipeline" button, at a specific time. It wouldn't make sense for .gitlab-ci.yml to say:

job:
  except:
    users: joe

.gitlab-ci.yml is part of the source code, and describes how the code is built; not whether or not it should be built at a specific time. Additionally, as presented in this issue, it is extremely un-flexible: One cannot select which schedules run which jobs.

My problem with the proposal given thus far is that it adds another seemingly unnecessary layer of indirection: "tags". We already have job and stage names; why can't the scheduler key on them?

Proposal

On the pipeline schedule page, there should be two regex fields: include and exclude. If a job name matches include it will be run; or if it matches exclude it will be skipped.

I think I would be equally delighted to have it implemented as you propose, @JonathonReinhart, rather than as my suggestion. For me, the important things to achieve is that some schedules can run some jobs and that others run others. (And that the commits trigger yet others.) You can put that coupling in the definition of the schedules, or the definition of the jobs, and the functionality is the same. I was just starting from the fact that except: schedules is already in the jobs (along with lots of other restrictions).

The only caveat is that under your proposal, you definitely have to keep only/except: schedules` available to the yaml file.

moved to gitlab#10662

Hey! 👋

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue