Add pipeline graph

Totally crap mockups.

List of pipelines:

Click on specific pipeline, see graphic visualization of pipeline including statuses, and list of build jobs with more detail than can be shown on graphic:

Click on specific job (via pipeline graphic or build list) to see build log and other meta data:

Would be nice to be able to compress complex pipeline into small vertical space. Possibly have expanded view when first click on pipeline, but when looking at build detail, the focus should be on the build detail not the pipeline graphic, so having that auto-compress somehow might work.

@markpundsack Thanks. Awesome work. We can also think about making pipeline view collapsible in build view, but this is a really nice idea to visualize where this build is in the pipeline.

@markpundsack Awesome ideas :)

@markpundsack:

cc: @jschatz1 since the concepts in the mockup will be Frontend's job to implement :)

Now that we've got 70 jobs for gitlab-ce, it's a great example to flesh out how we would visualize it. One thought is that we'd want to compress the parallel jobs into some concise image, yet still let you see individual status so you can find failures. I'm imagining one node on the pipeline, but with X boxes inside the node for each of the X parallel jobs. You could even do some kind of expanding thing where you click on the rspec node and then it pops up a bigger view of the 20 individual jobs. I really need to collaborate with a designer to flesh this out.

@markpundsack I'm looking forward to proposal. It looks terrible now :)

mentioned in issue #18516 (closed)

@markpundsack I love the idea of a new visual identity. I love that it's a rough drawing (I love balsalmiq) and it looks awesome. I am not sure I agree with the visuals. I'll try and come up with some good alternative other than just saying I don't like it.

@jschatz1 Awesome! Looking forward to seeing what you have in mind.

@hazelyang @annabeldunstone Would love your feedback as well.

@markpundsack now that @hazelyang is available I think she can handle it. @hazelyang can you provide some feedback here?

mentioned in issue #19324 (moved)

My proposed design, please let me know if you have feedback.

Pipeline

Build

Build - expanding graphic visualization

Not sure about reading a pipeline, but to view a particular build, the most important thing for me is the console output. It's definitely true for the failed build, and it's also very important for succeeded build, because I would like to see what's done for that build.

Pipeline chart is great, but not very important when I am reading a build. Builds list aren't very important either. So most of time I would just scroll down and read the console output, and then scroll back on top to check perhaps the other failed builds. Succeeded ones are less important.

So I think it's good to have pipeline chart collapsed as default, and expand it when clicked, and I should be able to navigate between builds with that pipeline chart, and then the old builds list are no longer needed. It really took up too much space and could be replaced by pipeline chart anyway.

@godfat Thanks for your feedback!

Pipeline

Builds with collapsed chart as default

If you want to go to other builds, you can expand the chart and then navigate through it.

Builds with expanding graphic visualization

@hazelyang Oh wow, that's awesome from my perspective :D

Thanks @hazelyang!

Even when there's only one job in a stage, the job may have a different name than the stage. Should we put job names under every check/fail icon, not just the ones with multiple jobs?
Putting the job name under the check/fail icon takes up a lot of vertical space, and requires even more blank space so that you can tell whether the words go with the icon above or below. What about putting the icon and words next to each other horizontally? This might allow us to support a flow with 20 or even 40 parallel jobs a little better. It might mean long pipelines scroll horizontally though.
Having to expand the pipeline view to navigate to another build feels awkward. If it's really most common to look through the failed builds, perhaps we could just show all the failed builds right there, and only need to expand if you want to see non-failed builds or the overall flow.
The snake flow of builds in the last expanded image doesn't make sense to me. These builds are all orthogonal to each other, run in parallel, not sequence. Why join them with lines? It's interesting visually, but not logically.
Just an idea, but following on the idea of showing all failed builds when one was selected, what about collapsing the pipeline to a single line, with dots for each stage, and when you click on a failed icon, it shows another row below with just the failed builds. Most of the time there should only be a few, but if you really have 20 failed builds, then it falls back to the wrapped build list like we have currently. The difference being that since we're only showing failed builds, most of the time the list will be small. Also, it'll only show jobs for a single stage at a time, which will help a little. We could still have the more interesting graph, with dots for each parallel job, as the pipeline overview, but not when viewing a build. One downside would be there's no way to jump to a succeeded build. Perhaps that could be a filter toggle so you could choose to see all jobs if you really wanted, but it wouldn't be the default.
In (5), maybe pending/running builds would have to show by default as well, so you could follow the progress.
Turning it into a single line might allow us to move the pipeline above the build meta data, which is better IA, and supports more horizontal space.

@hazelyang Where's the source for those images? I don't see them in gitlab-design.

@markpundsack I just added them to gitlab-design.

Pipeline page

I put the job name next to the icon and decreased the margin between each dots.

Builds page - A few of failed builds in a stage.

Builds page - If there are a bunch of failed builds in stage.

Thanks @hazelyang! So what do you think of that design of the pipeline page?

It's not as clean and simple as the first one, but does have a lot more information in it.

If we go with something like this, I feel like the build page should retain some element of the pipeline graphic.

I wanted to see what it would look like with gitlab-ce's pipeline. Here's my rendering. Admittedly, we're a bit of an exception. But for large projects, having 40+ jobs is common. It also highlights that maybe there can be a more compact view for the parallel jobs (rspec and spinach). I also tried throwing an oval around the job name. I obviously gave up connecting all the lines though. :)

Idea for compressing split jobs. Not sure what to do when X of Y jobs fail within rspec or spinach.

Showing collapsed pipeline with consistent visuals, and jobs below, limited to 1 line with More button following discussion on #18516 (closed).

And here's how it could look when looking at one build in a pipeline. I tried to make the IA work, where the build status was under the build selection. The sidebar still appears outside of the pipeline stages, so it's not quite right. But you get the idea.

Pipeline

Builds

Failed builds will be put first.

I tried to use collapsed chart as a tab bar, but it looks unclear, so I still use the real tab.

That works pretty well!

The final state looks a little weird with lines going nowhere. Most of the time there will be only one job in the terminal stage. If there are more than one, maybe they just there with no lines.

The 5/20 might be confused for 5 succeeded vs 5 failed. And it makes me wonder what it would look like if there were some failed, some running, some succeeded.

When something is selected from More, it needs highlighting (bold and underlined).

I think rspec and spinach are really a special case here. Not sure if we could really collapse them. We might want to introduce new configuration in order to collapse them.

Yes, the above assumes some kind of automatic parallelization, which we don't have yet.

mentioned in issue #19982 (closed)

Changed title: WIP: Render builds in context of pipeline → Render builds in context of pipeline

Milestone changed to %8.11

Added frontend label

Removed ~19492 label

Added ~122770 label

Added ~19492 label

@jschatz1 You previously said you didn't like the visuals of my mockup. How do you like these latest ones?

Changed title: Render builds in context of pipeline → Add pipeline graph and show builds in context of pipeline

Added ~307500 label

I think we could just show the total amount of builds in this page, so I dropped 5/20.

Yes, we have too many lines. I removed the line under Build #2429046 for commit ....

Removed ~307500 label

@annabeldunstone BTW, @jschatz1 suggested that 8.11 doesn't have a lot of ~Frontend commitments this release, so feel free to ask for help if you need to.

@hazelyang What do you think about the pipeline graph having lines after the last stage joining, but going nowhere? Can we do without the lines at all? Most of the time, I think people will have a single job in the final stage, but even if they have multiple, it might look better.

@hazelyang What do you think should happen when you click on a job in the pipeline graph? I'm assuming if it's a single job, then you go to the build for that job. But if the job is a parallel job, which build would you go to? Would you go to the first one? The first failed one? Or show a drop-down so you have to pick a job from a list?

@ayufan I kind of like your naming convention of jobname {index} {total} like you have for rspec on gitlab-ce. Do you think if we were to implement automatic parallelization, you might keep that naming convention? If so, then we could do something smart with it now and just detect that naming convention and collapse related jobs even before we support automatic parallelization. We could parse jobname {index}/{total} as well.

Edit: or of course we could just ignore that for this release, and have a giant graph for gitlab-ce.

@markpundsack

I tend to think that we should ship minimal viable change, and improve later. Collapsing thing fits into category to that we should work on after the first iteration. It will make the implementation and Frontend easier and then we can think what naming conventions we should support.

It should still be simple thing to do in second iteration and pretty likely to have that as part of 8.11, but having a bigger plan puts more risk on not shipping this at all. The first iteration will not solve our problem, but will solve problem of 95% of people when they do have easy pipelines :)

@markpundsack

I hope that you also included in your plan a play button (for manual actions) as part of the pipeline visualisation :)

Good point. We could put a play button instead of a pending icon, for jobs with manual actions. Not sure if that would look actionable enough. In other pipelines, the "play" action would be on the between the stages rather than on the job itself. If a stage has multiple jobs though, we need to associate the action with a specific job, not the stage.

@hazelyang Any thoughts?

@markpundsack

What do you think about the pipeline graph having lines after the last stage joining, but going nowhere?

I am not really understand what do you mean, could you explain more? Thanks!

We could put a play button instead of a pending icon, for jobs with manual actions. Not sure if that would look actionable enough. In other pipelines, the "play" action would be on the between the stages rather than on the job itself. If a stage has multiple jobs though, we need to associate the action with a specific job, not the stage.

Yes, we can add play button. Before we add it to the mockup, can you tell me why you think we should have play buttonin the graphic visualization? I am just wondering it probably make the graphic complex and people can do the same action in the table below the visualization.

What do you think about the pipeline graph having lines after the last stage joining, but going nowhere?

I assume it's:

Yes, that's what I mean. Thanks @godfat.

@hazelyang One reason to have the play button in the visualization is because it's cool. :) It's a great feature, and our competitors are doing it. :) OK, that's not a really good reason.

Real reasons:

It better shows the pipeline flow, so someone understands why one of the jobs hasn't run yet (because it's waiting for a manual action).
It allows completion of the pipeline from the overview without having to find the specific job in the list below.

If it degrades the graph too much, then let's not put it in. But my guess is that we can do it in a way that makes things even clearer, rather than just more complex.

Removed frontend label

Removed Product work label

Reassigned to @hazelyang

@godfat haha, thanks!

@markpundsack Thanks for explanation. I added play button to the mockup.

Have few questions:

There will be play buttons in passed stages or in stages of all status (passed, running, failed, cancel, pending)?
If the build is pending, there will have a play button as well?

@hazelyang The play button shouldn't show up until the stage before has succeeded. While the manual job is running, it shouldn't have the play action anymore. I'm a little torn whether a manual job should have the play action after it's already finished and succeeded, but for now, I think we let people replay the action as many times as they like.

I kinda envisioned having the play button before, or instead of, the pause icon for rubocop. Kind of like the play action was part of the flow lines itself. I have no idea how that would work though. I couldn't find up another example, but I swear I've seen one of our competitors put a play button in between the stages. That's not how our actions work, but it helps visualize that a manual step is needed to progress to the next stage.

Updated!

Thanks @hazelyang!

I'm not sure why you have the play button between Prepare and Test. There's no playable action in Test.

It looks weird having the play icon both between test and post test, and on rubocop.

A more realistic example would have another stage after "Post Test" with a deploy job. Perhaps if the inter-stage play button only appeared if there were only a single playable job in the following stage? And then use a different icon for the playable job itself? Of course at that point, maybe just using the play icon for the job and dropping the icon between stages is fine. Not sure.

Do you think the play icon is clear enough that it is a clickable button, whereas the other icons are just showing status?

Oh, also just realized that if people are using the graph for navigation, e.g. to click on jobs in the graph to go to the details for those jobs, they might click on the play icon by mistake, intending on just viewing the job rather than triggering it.

I did some changes on visual design to make jobs look like more clickable and clean.

I'm not sure. I can't put my finger on why, but I liked it better without the boxes. (Also the converging lines in both directions between stages.) I realize it was probably necessary to make the play button work though. @hazelyang, got any other ideas?

I proposed two designs. I thought B is cleaner and easier to understand the jobs are clickable than A. If we allow users to click the jobs and go to the build detail pages, maybe make the job look more like a button would be better.

A

B

B does look cleaner, and works well overall. Interestingly, the Deploy job in A stands out more as actionable because it's the only one with a bounding box, and because it's shaded differently. But I think the button on B's Deploy job works too, and perhaps especially because the Deploy job doesn't have an icon to the left of it.

What do you think @annabeldunstone ?

Option B looks great @hazelyang! I agree with @markpundsack though- maybe we could style the Deploy button to make it stand out from the other test names. The gray background from option A might work well

Thanks for the suggestion. Looks good.

`Deploy` button with grey background.

That's surprisingly good!

What do you think we should do with retries? Should we hide them from the graph, but show them in the job list below?

mentioned in issue #20638 (closed)

@hazelyang is this ready for me to start working on?

From https://gitlab.com/gitlab-org/gitlab-ce/issues/20638, I wonder how we should handle skipped jobs other than manual actions. For simplicity, we should start by just showing them including the skipped icon. Anyway, my current thinking is that the pipeline graph should show all skipped and future (created) actions, but the build list below the graph should only include jobs that actually ran. The converse is that the list below should include retries, but the pipeline graph would only include the most recent of each job. Or possibly include retries in a drop-down, just like shown for rspec.

Also, the logic of when: on_failure, when: on_success, and the eventual flow around dependencies implies that maybe our simple linear pipeline will need to evolve to include directed graph or flowchart-like flow where a stage has branches for success/failure of the previous stage, and possibly skip stages altogether. e.g. a run on master might skip the staging stage and jump right to a production stage. Rather than showing that as the staging deploy being skipped, it could show a curved line routing around the skipped job going directly to the production deploy job. Although, now that I write that, I think the current design actually does work well, if you consider that each job may be optional. It is technically correct to say that one stage triggers the next one, with optional jobs that are skipped, and then the flow continues to the next stage. As we define things today though, a stage/job that is ignored because of only and except conditions won't even be created, so won't show up in the pipeline graph at all. So the graph will look different for different pipeline runs.

@annabeldunstone FWIW, from my perspective the design is good to go.

@annabeldunstone Please do it. Thanks.

Added frontend and removed ~19492 labels

Reassigned to @annabeldunstone

mentioned in merge request !5742 (merged)

@hazelyang instead of expanding/minimizing the graph, we're thinking it might be better to completely hide it/show it. Do you have any ideas on how that should look (both hidden and expanded)?

Totally agree on just hiding it.

But, just to throw this out, could we hide the jobs, but leave the stages? Perhaps then add the state icon and rounded oval around the stage names themselves? I don't know if that will look cool, and feel like a compressed version of the pipeline, or look jarring because we'd be changing the treatment of the stage names.

If we did, then clicking on a stage node could show a drop-down with all the job names, like we've already planned for the rspec-type parallel jobs.

Expand graph

Hidden graph

mentioned in issue #19793 (closed)

mentioned in issue #20800 (closed)

mentioned in issue #20863 (moved)

Changed title: Add pipeline graph and show builds in context of pipeline → Add pipeline graph

Updated title and description to reduce scope, focusing on the pipeline graph. Moved showing builds in context of pipelines to #20863 (moved).

Going to use the FA play icon for the meantime, but when you get a chance, could you create an SVG @hazelyang?

Status changed to closed by merge request !5742 (merged)

@annabeldunstone Added it to production/_assets/svg.

Thanks @hazelyang!

mentioned in commit 26730316

mentioned in issue #21286 (closed)

mentioned in issue gitlab#5799

mentioned in merge request !14785

Add pipeline graph

Description

Proposal

Pipeline:

Todo

Links / references

Designs

Child items ...

Activity

Pipeline

Build

Build - expanding graphic visualization

Pipeline

Builds with collapsed chart as default