Visual editor for `.gitlab-ci.yml`

Description

Creating/editing .gitlab-ci.yml files is getting harder and harder. Perhaps a visual editor would help?

Proposal

Nothing specific yet. It could be as simple as a visual interface to the literal YAML file, but I think a good implementation would go beyond that and hide a lot of complexity. Perhaps present the information in completely different ways. Like a checkbox to enable auto deploys, which then generates several lines of YAML. Also provide discoverability of advanced configuration options.

Links

• Building on #20785 (moved)
• Blue Ocean Pipeline Editor: https://github.com/jenkinsci/blueocean-pipeline-editor-plugin

mentioned in issue #21624 (moved)

Grafana's query building is an analogue, although not a direct one. Similar sort of challenge, good to mine for ideas.

Mentioned in issue #20785 (moved)

Mentioned in issue #21624 (moved)

Mentioned in issue #24913 (closed)

@markpundsack @joshlambert as an impression, what about:

in corporation visual mini snippets, with a switch to view actual code and a visual expandable GUI with dropdowns and buttons etc

drag-and-drop interface from left to right

inspiration framer code folds: https://d13yacurqjgara.cloudfront.net/users/1005/screenshots/2767491/dribbble-c.gif

@dimitrieh I think yours looks pretty good! I wonder if though if we couldn't make the job and stage separation a little more clear? I apologize for the eyesore, but tried to mock up what I am trying to explain. (I promise my paper version looked better, and clearly should not quit my day job.)

In the spirit of brainstorming, it has a lot of various ideas in aside from the main method of visualizing the flow. Some are probably bad, but hopefully a few may spark more discussion.

• A more card based approach, with each stage being a swimlane and a job being a card in that swimlane. This seems a little more clear to me, and can also help illustrate other ideas later (like flow for a given branch/tag).
• Inline linting, which runs automatically. Since linting doesn't seem very computationally heavy, can we just run this all the time as a user works?
  • Can we do autocomplete based on what is valid within that given stage, job, or specific command?
• A template library, which can have templates for common stage types for each language.
• An "Attribute/Inspector" pane, which can provide some insight into what you are currently working on. For example:
  • Can show you what DockerFile this stage will use, in case you forgot to override from the defaults
  • A merged script, showing the complete picture inclusive of before_script and other types of options.
• A command which could take as input a given branch/tag name, and illustrate the stage/job flow. (To help debug when/only/etc type problems?)
• A button to execute a given job in a runner, in real time. This way you can debug fully in the Visual Editor, without having to move to your local machine.
  • You still might end up having to do that for more complex problems, but this seems like a nice quality of life improvement.
  • Just give it a branch/tag and away it goes.

All good ideas. I like the card-based drag-and-drop for creating/arranging a pipeline. And snippets have their place too. Perhaps within a single job, although that's not strictly true. Other vendors have done things with dragging-and-dropping just portions of the configuration, like dependent services such as adding Postgres to your test.

Unrelated to any of the above, the last couple of days I've been envisioning one of those children's books where there are three sections, top, middle, and bottom, that each flip independently. Like one page has a robot, and another has a person, but you can flip only the top section, and switch to the man's head on a robot body and legs.

Anyway, I'm thinking of build, test, and deploy as being pieces you can choose independent of the other choices. Like Build a Rails app, test with rspec, deploy to Heroku. Or build a docker image, test with npm test, and deploy to Kubernetes. Or any permutation of the above.

Or going beyond that, being able to add review apps to an otherwise complete build, test, deploy flow. Or being able to pick between automatic or manual deploy to production.

I'm sure this is too simplistic, and falls down in many ways. But feels incremental from our existing template dropdown, and getting us towards more of a snippet or card approach.

At any rate, I think this topic will be really hard to do. I think it would be easier if we start with visualizing the configuration of an existing pipeline, heavily leveraging our current pipeline graphics. Then maybe extend that. Perhaps with the ability to visualize the pipeline for given branch/tag or other circumstance. Or maybe that won't be necessary if we augment the pipeline visualization well. Sure, it currently doesn't show only, except, and when clauses, but maybe there's an easy solution for that.

Then maybe orthogonally, improve the text-based editing, with autocomplete, etc. Then add snippets (text-based only). Add a live-preview of the text editing.

Anyway, that's only one path forward, but it feels more achievable than jumping directly into a GUI pipeline builder. And more leveragable by intermediate users as invariably simple GUI builders are most valuable for beginners. And it fits in well with our incremental approach to everything.

Other prior art: https://github.com/jenkinsci/blueocean-pipeline-editor-plugin

@dimitrieh I think having raw node addition/subtraction is probably a good place to start. But building on the templates idea, I'm picturing a gallery of node collections that you could drag/drop into your flow to add functionality to your CI/CD configuration. Like a template for review apps that includes two nodes, start_review and stop_review. Or a continuous delivery pipelet that includes a job to autodeploy to staging with manual job to deploy to staging. And a continuous deployment pipeline that goes straight to autodeploying to production. Maybe dropping the latter on top of the pipeline replaces the former because they're mutually exclusive. I have no idea how to encode that. And maybe dragging/dropping the pipelet onto the pipeline is overkill when you could just select the choice from a couple radio buttons. But maybe for other cases, it's more additive. Like adding rubocop to an existing pipeline.

Maybe it's as simple as having the template define whether the jobs contained in the template can coexist with other jobs in the same stage or not. So if you drag CDeployment over CDelivery, it knows it has to replace those stages. (although CDeployment wouldn't have a staging stage, so maybe it would also have to define it as blank...

@markpundsack just as a note here:

Buddy works does a pretty good job here, all from the new pipeline button

related: https://gitlab.com/gitlab-org/gitlab-ce/issues/27795#note_23069813

I think we could do a similar thing, also for

• configuring notifications (in app vs email)

• selecting docker containers

• selecting other tools than the kubernetes cluster, like https://gitlab.com/gitlab-org/gitlab-ce/issues/27897

• configuring auto-deploy as a template, while each of thes sub actions can be like mini snippets

If we combine this idea (just brainstorming here) with a visual representation similar to our pipeline graph, we may come a long way

@markpundsack from a user perspective thinking along similar patterns like, setting up a pipeline, instead of setting up gitlab-ci.yml file makes much more sense and has the ability to make this a lot more approachable!

We still save it as a file within the repository and let the user know that if they are interested in editing the file manually.

This would mean that if we recognise certain patterns in the ci.yml file we can entertain our users with a gui version of that snippet.

perhaps do a meeting on this sometime soon

@dimitrieh I agree, that was sort of the direction I was trying to go with the "card" approach. You would add a stage, job, etc. and you could visually see how it would fit in the flow. Then of course you could go in and edit that particular section. Buddy looks to have gone even farther though and each "action" seems to have a specific UI for it. Not sure we need to go to that level, though.

mentioned in issue #27897 (moved)

Yep, this is a good direction. Buddy has a crazy amount of configuration for that template, but I can see the appeal.

@dimitrieh One thing we haven't figured out yet is how to indicate only, except, and when. For currently rendered pipelines, these are all processed before the pipeline is created, so there's no need to indicate them. But for a graphic view/editor, this will become important. Rather than trying to indicate them via lines, I'm thinking of just adding it as meta information for each job. So some chrome or adornment on the job oval, maybe with the content hidden until hover/click.

If a bunch of jobs have the same only/except conditions though, it might be nice to group them in a stream. It could then look like parallel pipelines, because that's essentially what they are.

mentioned in issue #22558 (closed)

jenkins update in this regard

Sounds like we all agree, this would be a great feature. The big question is when? Or some iteration of the card approach first per @joshlambert example. @eliran.mesika and @mbell do we want to share another very large customers request for this feature.

mentioned in issue #32638 (moved)

added moonshots label

mentioned in issue #33165 (closed)

mentioned in issue #14266 (moved)

mentioned in issue gitlab#5770

moved to gitlab#5974

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

mentioned in issue gitlab#6032