Cycle Analytics: second iteration

Mentioned in issue #21934 (closed)

@regisF which issues / events do we show? Most recently completed in this step (this means that this doesn't contribute yet to the cycle analytics time as showed, right?)? Most recent of this event that has completed the entire cycle (then why would we need multiple lists?)? What changes if we change the range from 30 to 90 days?

which issues / events do we show? Most recently completed in this step (this means that this doesn't contribute yet to the cycle analytics time as showed, right?)? Most recent of this event that has completed the entire cycle (then why would we need multiple lists?)

@JobV I don't understand your point.

Cycle Analytics will only show what's been pushed to production anyway. That means we start from the end: we see what's been pushed to production, then check which issues started this cycle. That means, we can have multiples issues that have led to something being pushed to production in the same range.

Each issue is followed by a dedicated commit, MR, test time, etc... Hence multiple items in each stage, again.

Considering this, each stage will show the totality of the events in it - what is used today to calculate the median. Let me know if I'm completely off here.

What changes if we change the range from 30 to 90 days?

The entire table is being recalculated with everything that has been pushed to production in the last 90 days, as it is now.

To replace the percent 95th we could put the total estimated time, that is the product between the delta and the number of items, what do you think?

Just show the mockups to get the feedback. 🙂 I just want to check I am on the track, and then I can do other screens or modify them.

Issue

Plan

Code

@hazelyang we are definitely on track!

I think there's some confusion around "Ability to see events in each stage" vs showing X recent ideas that have been shipped to production. This was discussed on the original issue, but could use some more discussion, and even better, actual and target customer feedback. I'd like us to understand what the customer actually wants here, or what would benefit the customer most.

Related: Personally, now that I've seen this live for a few projects, and seen that some projects have completely empty analytics because they don't deploy to production (I'm looking at yougitlab-ce), I wonder if it's worth re-evaluating that decision too. Wouldn't it be more helpful to show people data for every stage they do have? It's a different type if analytics. e.g. Independent statistics vs cohort analysis. But may be just as valuable (and certainly more valuable for people with zero analytics today). That might make it easier to reason about the above question about seeing "events in each stage".

One obvious complication is that it's hard to know how many ideas haven't been turned into issues yet. And that statistics will be skewed for issues that never turn into code.

But I guess the overall message here is that now that we have an iteration out there, let's talk to customers and see what to ship next rather than following the design from before we had much customer feedback.

Sorry if this is off-topic, but I didn't see this anywhere: Is an MR counted as being "in review" if it's a WIP? In our workflow, we create the MR right away after the first commit, and tag it as WIP, so having that count as "in review" would muddy the metrics. Hopefully the reviewing time is time between the last time the WIP label is removed and the MR being merged.

Empty State

We can explain why there is no data and how to make it happened. If you have the better copies, please let me know. 🙂

@sytses we have a few concerns for the second iteration:

First of all, we haven't received a lot of feedback from the community for this feature. So it's a bit hard to figure out what the community wants at the moment. However:

One user has expressed a will to use Cycle Analytics even if they don't use the Deploy to production stage.
We also don't use our own tool to ship to production, so Cycle Analytics for gitlab-ce and gitlab-ee does not show any data.

As @markpundsack said above, "Wouldn't it be more helpful to show people data for every stage they do have?", not necessarily what's only been shipped to production? That way, every project will find value in this dashboard.

Can you tell us what's your opinion about this?

@regisF I think that would be great

Mentioned in issue #22579 (closed)

Just to provide some more feedback: The feature itself sounds great and I can't wait to use it. However in the current state it is not very useful. Making the analytics available for parts of the whole cycle would definitely help.

I would also like to point out my other issue #22581 (moved) - This could help to bring cycle analytics to a lot of people with several smaller repos. I just began transfering one of my projects (CxxProf) to Gitlab. The group will contain of about 5 to 10 repos in the end. Having Cycle analytics available for each repo separately will not provide much usable feedback. Having it available for the whole group would be a lot more useful.

At work we've set up a Gitlab instance with ~150 repos. Each repo contains a small component. Cycle Analytics does not provide anything useful at this stage. However being able to get the data for the whole repo or for repos where a specific user is participating would turn this around 180°.

The new proposal is therefore:

We need to measure everything that happened in the given time range, not only what's been pushed to production.
The difference with the old flow is that the first commit that we track, should not be linked to something that has been pushed to production - in the given time range.
All stages except staging and production will basically remain the same.
The staging and production stages will only take into account the issues that have been pushed to production.
We should make sure that the explanation on the last two stages (staging and production) are clear.
The "events" per stage stay the same as previously defined.
- Issue: list of issues created in the last XX days, that have been labeled or added to a milestone.
- Plan: list of commits that reference for the fist time an issue from the last stage.
- Code: list of MR created in this stage
- Test: List of unique builds triggered by the commits.
- Review: List of MR merged
- Staging: List of deployed builds
- Production: list of issues with the time from idea to production

@timothyandrew can you tell me if this makes sense to you?

@monsdar I understand your point, thanks for sharing.

I wonder though how unique your use case is compared to general use of the projects and repositories. I think we should work towards having a relevant Cycle Analytics at a project level first, then see how we can expand this to a group level. What do you think?

I agree - it's probably the easiest to first get it right for single repos and then find a way to scale it up to multiple repos at once.

I'm not sure how unique my way of doing things is. I can imagine that bigger projects that make use of the group-feature could be interested in aggregated cycle analytics. I guess there are differences between OSS and Enterprise too.

Test

Review

Staging

Production

If you have any feedback, please let know. 🙂

@regisF I am going to design the tooltips. Do you have the copies that you want to show in the tooltips?

@hazelyang just to nitpick: the Related issues title at the top, should change depending on the selected stage.

@axil @marcia can you help in providing relevant tooltips for the headers of this table?

just to nitpick: the Related issues title at the top, should change depending on the selected stage.

@regisF Thanks for your reminder! 😄