Merge Request Page - Results

marked the checklist item Share videos and research findings in a new issue as completed

changed the description

@dimitrieh The first user videos are back for the merge request page. The prototypes seem absolutely fine. There was one slight glitch with the Discussion tab for User 1 which you can see at 15m 7s into the video. Overall though, so far, so good

changed the description

Awesome @sarahod I have watched 1 and 2. I'll jot down my findings below and some assessment points for us to improve the script.

Script assessment points:

Can you share the script in the description? as it needs some alterations :) (its this one right?
Alteration: First question about the version.. is pretty unclear (even to me) .. especially as we are on the about.gitlab.com website instead of within gitlab.. what is the purpose here exactly? :)
Alteration: When was the last recorded amendment made to the repository? We should i think be clearer we are referring to the branch this MR is based on. So what about: When was the last recorded amendment made to the branch of this merge request?
Alteration: What is the current state of this merge request? is nice to know if the user knows its open/closed/merged.. but i thought we wanted to see if they get that the MR-widget details the advanced status :)... perhaps an extra question in this regard is necessary.. wdyt?

Findings 1

prototype 1:

scroll up/down -> doesn't notice the tabs (as they are not visible initially when opening the page.. imo)
branch indicator is not obvious enough (marked it as a 3)
didn't mention mr widget as part of status of MR question
user indicates later on that the changes tab isn't not visible initially, makes the task harder!
User expects/looks for "which stages" question in the tab row (which he discovered)... but misses the MR widget completely
Two arrow icon skipped icon is not clear to user
Small glitch in prototype switching to discussion tab (as mentioned )
Expects to see MR merge at the bottom (like GH) - idea here. to duplicate the mr widget contents there. - Misses MR widget completely again - fails task!

prototype 2:

Indicates that its really nice to immediately see the MR tabs.. (it's now always there)
Clearly notices merge status
Gets that you can extend description
Finds the merge button (which is disabled) :)
hasn't said anything about the changed branch indicators
hasn't discovered or specifically said anything about the status icons in the merge status tab (@sarahod what about a question, for the user to get to know what the merge status is without changing to the mr status tab?)
Prefers 2nd design

Findings 2

prototype 1:

User looks for description in discussion?
Finds mr widget, knows that they are tests
thinks amendment refers to any change to either code or discussion or system note (@sarahod should we improve the question?)
Expects branch name to be at the top
Thinks branch indicator is not clear enough, cannot see complete branch name (no tooltip in prototype)
Clearly more familiar with MR's compared to 1
"which stages" is not clearly defined (@sarahod perhaps we should refer to CI stages?)
Cannot see names of stages (not in prototype... perhaps we need an improvement here)
Notices the closing of issue mention in the MR widget

prototype 2:

Notices new branch indicators, likes the branch icons!
Notices the status indicator of MR status in the mr status tab... is confused by it ..as now there is double information available.. as the MR widget is active, because of this.. its confused with the pipeline stages.. still notices its a bit different though (improvement necessary here: Only show icons, when the tab is not active. wdyt @sarahod )
Gets the description extender
Confused if merge locally mention at bottom of mr widget, the same as "merge locally" button
Appreciates the clear view of MR-widget in a tab.
Admits he uses discussion tab less... thus widget being the default is a good one.
Prefers 2nd design

This is awesome @sarahod .. so excited haha!

assigned to @dimitrieh

changed the description

@dimitrieh Thanks for your comment! It was great to read your observations

Can you share the script in the description? as it needs some alterations :) (its this one right?

Correct, I've added it to the description. I've tweaked it based on my comments below. If I've misunderstood anything, feel free to just stick a comment on the Google Doc and ping me (if easier).

First question about the version.. is pretty unclear (even to me) .. especially as we are on the about.gitlab.com website instead of within gitlab.. what is the purpose here exactly? :)

I think it's interesting to know what version of GitLab users are using so we can gauge their familiarity with the existing MR experience. For example, User 2 is on 8.2, when he views the existing experience, the first comment he makes is "the interface has changed a bit". Therefore he doesn't have the same level of familiarity as somebody on 9.2 for example.

The first tasks, although verbal responses only, still require a landing page (constraint of the testing platform). It can be any URL you want, but I typically like to start with something that is unrelated to what's being tested to reduce bias (hence not placing them inside GitLab) but still makes them feel that they're in right place to start testing.

In the past, I used to place users on a generic, completely random landing page, such as Google, but occasionally a person would panic that they're not viewing the right thing (in their mind, they think they're testing a prototype for X company, so why are they viewing Google? Something must have broken!!). As I'm not there to intervene and re-assure them, I try to pick something representative of the company - in this case GitLab's homepage seemed like a reasonable choice. It's representative of GitLab but unrelated to what's being tested.

Having said this, after watching User 1, I did immediately want to move the question! But User 2's video came in before I did, which made me feel less worried I'd still like to capture this information but wonder if the question would be better at the end of the testing. It could be a text answer, where the user just types in what version of GitLab they think they are using. What do you think?

Alteration: When was the last recorded amendment made to the repository? We should i think be clearer we are referring to the branch this MR is based on. So what about: When was the last recorded amendment made to the branch of this merge request?

Good suggestion. That's fine with me.

Alteration: What is the current state of this merge request? is nice to know if the user knows its open/closed/merged.. but i thought we wanted to see if they get that the MR-widget details the advanced status :)... perhaps an extra question in this regard is necessary.. wdyt?

Sure! We can add an extra question, no problem. Do you have any thoughts around the question wording? Let me think how we can ask this...

User 1 - Prototype A

scroll up/down -> doesn't notice the tabs

Good observation. The user first interacts with the tabs after 11 mins - that's a long time.

User 1 - Prototype B

hasn't discovered or specifically said anything about the status icons in the merge status tab (@sarahod what about a question, for the user to get to know what the merge status is without changing to the mr status tab?)

Hmm I'm not sure I understand what you mean? The merge status tab is open by default, so they'll always see the merge status?

User 2 - Prototype A

thinks amendment refers to any change to either code or discussion or system note. (@sarahod should we improve the question?)

I think your suggestion above of When was the last recorded amendment made to the branch of this merge request? makes this question clearer.

"which stages" is not clearly defined (@sarahod perhaps we should refer to CI stages?)

Like this? What CI stages of the merge request have been passed?

Cannot see names of stages (not in prototype... perhaps we need an improvement here)

Yes, I'm wondering whether we need to do this too. For the first two users, if they could correctly identify where they thought they would find this information, then I've passed them on the task. I thought it was a bit unfair to only pass them if they could name the stage. Let me know if you want to tweak the prototype and I can pause the tests.

User 2 - Prototype B

Only show icons, when the tab is not active. wdyt @sarahod

Yeah potentially a good idea. I think it's only one user though that has said this so far, so I'd wait for further validation (keep the solution in your mind though!).

.. so excited haha!

Me too haha!

It could be a text answer, where the user just types in what version of GitLab they think they are using. What do you think?

@sarahod sounds good! :) the thing here is, that the user thinks he has to find the answer somehow on the current page he/she is looking for. As that is obviously not what you are after.. we should rephrase the question slightly to for example: Based on the current looks of the interface, which version of gitlab do you think you are using? This makes of course no sense.. if the user is viewing the about.gitlab.com homepage.. as that is not part of the application.

Sure! We can add an extra question, no problem. Do you have any thoughts around the question wording? Let me think how we can ask this...

What about Aside from this MR being open/closed/merged, what other more advanced status does it provide??

Hmm I'm not sure I understand what you mean? The merge status tab is open by default, so they'll always see the merge status?

Let this go :) This can be considered part of my answer above. What I am aiming for is an improvement to the prototype:

Only show icons, when the tab is not active. wdyt

as can be read in https://gitlab.com/gitlab-org/ux-research/issues/21#note_34583235

However I see you already answered this:

Yeah potentially a good idea. I think it's only one user though that has said this so far, so I'd wait for further validation (keep the solution in your mind though!).

I however want to pursue this rather sooner than later, as I think this should be part of the MVP. Mispresented information is bad information.. let's fix that to get results we can work with. wdyt?

Let me know if you want to tweak the prototype and I can pause the tests.

Yes, i think this is necessary due to the question asked. I'll add it as soon as I get your answers to this comment.

Like this? What CI stages of the merge request have been passed?

Almost, i think: What pipeline stages of the merge request have been passed?

Thanks @dimitrieh, all sounds good!

I've paused the tests so you can make the necessary changes to the prototype. I'll update the script in Validately too. Let me know when you're done and I'll resume the tests. Thanks!

@dimitrieh is there any way you can clone the prototypes and make the changes mentioned above on new URLs? I've got a couple of users waiting to complete the tests but I don't want to resume them if you're still working on the prototypes. Thanks!

What pipeline stages of the merge request have been passed?

I've realised that we can't phrase this question in this way - it's too leading. Users will just look for the word pipelines in this interface. Essentially they'll be matching words and we won't know whether the pipelines tab is where they would naturally look for this information.

Aside from this MR being open/closed/merged, what other more advanced status does it provide?

This is a similar issue. We use the word status on the merge status tab.

I'm currently uploading a third user video to this issue (which was completed yesterday just before I paused the tests). The user does refer to the widget when answering the question of: What is the current state of this merge request? , so it's possible that we might be being a little too hasty by adding an additional question.

There's also a potential overlap with: What stages of the merge request have been passed? because a user could use the widget or the Pipelines tab.

changed the description

as discussed in chat, I'll be updating the prototypes today. So testers can finish on friday, before the weekend (and before i'll be on holidays ) expect an update tonight!

mentioned in commit gitlab-design@b970a7ce

@sarahod prototypes are ready again! Implemented:

added tooltips for mini-pipeline-graph stages
made the mini-mr-widget inside the tab disappear when the tab is active, and reappear when the another tab becomes active (this was actually quite hard to do with my non flexible code haha)

@dimitrieh awesome news, thanks!

changed the description

@sarahod @sarrahvesselov will stay assigned, even though i'll be away for some days. When I get back I'll explore and write down my findings of the rest of the videos here. What is the timeframe in which we'll be able to act upon the results discovered here? %9.6 ?

cc: @victorwu

changed the description

marked the checklist item Watch videos [11/11 completed] as completed

marked the checklist item Pay participants [11/11 completed] as completed

changed the description

marked the checklist item Analyse videos [in progress] as completed

Testing complete! The full report is available above.

I've added my recommendations under Summary of findings but feel free to disagree. The decision is not mine alone

I'll run through the summary tomorrow in the UX Research update. We can also have a more in-depth de-brief in the UX team call (however we may want to wait for @dimitrieh to return from vacation).

@sarahod @sarrahvesselov will stay assigned, even though i'll be away for some days. When I get back I'll explore and write down my findings of the rest of the videos here. What is the timeframe in which we'll be able to act upon the results discovered here? %9.6 ?

I guess this depends on the scope of the changes you feel you want to make

changed the description

made the issue visible to everyone

added Done and removed In Progress labels

This is excellent @sarahod! I think we can talk it over in the UX call, we won't be able to move forward on anything until @dimitrieh returns.

marked the checklist item Summarise findings in UX team call as completed

@annabeldunstone Sarah said you might have had some idea's as in how to improve the mr view. Can you write them down here? :) Would love to have a look into your thoughts on the subject. Optionally also what you yourself really liked from the research done above (see the prototype URLs)

We saw lots of users looking for the Merge CTA near the comment box within the Discussion tab. Is this a better location for it?

this is purely because github has it located there

@sarahod what do you think about:

we should perhaps look into ways into making the MR widget more compact ;), but in any case, the tabs should now always be above the fold.

description moved to discussion tab, + various other changes (takes some cues from https://www.pmease.com/gitplex )

cc: @sarrahvesselov @tauriedavis

mentioned in commit gitlab-design@ec492dc2

@dimitrieh A few initial thoughts:

Moving the description to the Discussion tab - Smaller teams didn't typically utilise the Discussion tab on MRs - they verbally communicated with one another or used a chat client to discuss progress. I have some concerns that they wouldn't naturally look in the Discussion tab for the description as it's not a feature they're currently utilising.

In relation to the above, how does the design of MRs impact issues? I ask because I wonder whether there needs to be consistency between the content layout of issues and merge requests? i.e. Would the description move to the Discussion tab on issues as well? Our latest research (#27) highlighted that there are users who, similarly to merge requests, do not utilise the Discussions tab on issues. Obviously, this may never be a problem if you don't envisage MRs/issues using the same/a similar content layout.

There were some comments from users in reference to what the Discussion tab is called:

Users felt that the word 'Discussion' didn't accurately describe the content within the Discussions tab. One user stated that it is a "thread of activity". He also pointed out that we use the word 'Comments' within the Changes tab, subsequently, he felt there was some inconsistency in how we've chosen to label similar content. A second user stated that everyone uses "comments" on the web, rather than the word "Discussion".

I think by moving the MR description into the Discussion tab, we may want to review what this tab is called.

Would you still hide/expand the MR description within the Discussion tab? If the description is long, it may be a pain for users to scroll down to see the latest comments within the Discussion tab. Users may also want to read the latest comments whilst referring to the MR description - this would mean scrolling back and forth within the tab.

this is purely because github has it located there

Exactly, but 50% (6/12) of users expecting to see the Merge CTA near the comment box within the Discussion tab is still very high. I'm not advocating that we copy GitHub, but maybe we should test something similar (it doesn't have to be the only prototype/idea we test).

@sarrahvesselov as I am decoupling from the mr widget stuff for now, I will unassign from this. I feel we can do more with the information presented here, when https://gitlab.com/gitlab-org/gitlab-ce/issues/37479 is completed (although this is more focused on the structuring of the entire mr view of course)

unassigned @dimitrieh

Thanks @dimitrieh!

@sarahod, I have asked Dimitrie to focus on the portfolio management efforts for the time being. Are there some actionable items we can take away from this research to keep the momentum going?

@sarrahvesselov The recommendations outlined in the description really need some design resource and then re-testing. I think @dimitrieh is right, tackling https://gitlab.com/gitlab-org/gitlab-ce/issues/37479 first would probably help tackle the problems this research has highlighted.

The new style of branch name (as per prototype B) tested well, perhaps that could be implemented?

The new style of branch name (as per prototype B) tested well, perhaps that could be implemented?

That sounds like a nice step forward. Can you create an issue for that and indicate it came as a result of this research?

100% agree on your other points.

Sarah said you might have had some idea's as in how to improve the mr view. Can you write them down here? :) Would love to have a look into your thoughts on the subject.

@dimitrieh sorry for the super long delay! But I'm really excited about all the work you guys are doing for the MR view, since in my experience it's where people spend a lot of their time.

There's one thing I was hoping for that's still missing from mockups, and that's some sort of container around comments in the discussion tab. Currently everything is sort of floating around in rows with only single gray lines to separate comments. I was playing around with this a while back, and most of the feedback seemed to be centered around its similarities with Github. It's a fair point, but I don't think it should be the only reason not to do something.

Optionally also what you yourself really liked from the research done above (see the prototype URLs)

I love that we're doing something about the MR description length. It would be amazing to open an MR and have the tabs in the exact same place, regardless of content. I had originally opened an MR to move it into the discussions tab, but quite a few people pushed back because they like to see the description no matter what tab they're on. I think the max-height is a good compromise. A question about implementation though- do you think we should always show the collapsed version? And have the user click to expand for every MR? Or maybe add something in user preferences for those who always want to see the full description?
Glad to see the pipeline status moved out of the widget, to help de-clutter some. I agree with what was said earlier, that it shouldn't be the default tab, and I'm not sure the entire pipeline is necessary in the tab. What if the pipeline has 10 stages? Maybe just the current pipeline status icon somewhere on the page would be sufficient.

If we do change comments to be encased, similar to Annabels mockup, please ensure the design still works with the designs for discussions:

https://gitlab.com/gitlab-org/gitlab-ce/issues/30299 https://gitlab.com/gitlab-org/gitlab-ce/issues/32451 https://gitlab.com/gitlab-org/gitlab-ce/issues/32452

User Number	Job	GitLab User?	Prototype Order	Video
1	Unrelated - Uses GitLab for personal projects	Yes	AB	http://bit.ly/2uyzHWb (Restrictive access, user does not wish to share the video publicly)
2	Software Engineer	Yes	AB	http://bit.ly/2sOOKh9
3	Researcher	Yes	AB	http://bit.ly/2t6ClFu
4	Software Engineer	Yes	BA	http://bit.ly/2sYLeN6
5	PhD student	Yes	BA	http://bit.ly/2tnetZI
6	Associate Director (Team Lead and participates in software development)	Yes	BA	http://bit.ly/2tUsVds
7	Mobile Developer	No, BitBucket & GitHub User	BA	http://bit.ly/2tVMz90
8	Software Developer	No, BitBucket & GitHub User	BA	http://bit.ly/2u0ULEM
9	Summer Internship at Software Security Firm	No, BitBucket & GitHub User	AB	http://bit.ly/2t7hRIz
10	DevOps Engineer	No, BitBucket & GitHub User	AB	http://bit.ly/2uurnJG (Restrictive access, user does not wish to share the video publicly)
11	Senior Front End Web/Mobile Developer (Team Lead)	No, BitBucket & GitHub User	AB	http://bit.ly/2uDZ4cv

User Number	Prototype Order	Preference
1	AB	B
3	AB	B
4	BA	A
5	BA	A
6	BA	B
7	BA	NIL
8	BA	NIL
9	AB	B
10	AB	B
11	AB	B

Task	Success Rate	Success Rate	Satisfaction Score	Satisfaction Score	Time On Task	Time On Task
-	A	B	A	B	A	B
Branch	5/6	5/5	4/5	5/5	15s	6s
Status	5/6	5/5	5/5	4/5	6s	2s
Stages	5/6	2/5	3/5	4/5	18s	21s
Merging	5/6	3/5	4/5	3/5	16s	31s
Issue Number	6/6	5/5	5/5	5/5	3s	3s

Area Affected	Problem	Number of Affected Users	Recommendations/Further Comments
Merge CTA	Expected to see the Merge CTA near the comment box within the Discussion tab.	6	GitHub's Merge CTA is positioned above a comment box, hence user behaviour to look in a similar location.
Tabs	Users failed to interact with the tabs during the testing. For some users, the tabs just didn't grab their attention and they scrolled past them multiple times.	4 (Prototype A: 2/6, Prototype B: 2/5)
Iconography on Merge Status tab	Users did not understand what the icons represented. One user commented that it would become even more confusing if there were multiple pipelines on one build.	3
Iconography within the Pipelines tab	Similar to the above, users felt that the icons used within the Pipelines tab did not provide them with enough information. Even when hovering over the icons, it wasn't clear to them what was happening beyond that something was failing. One user described the 'post-test' icon as a "fancy embellishment".	3
Timestamps within the Discussion tab	Would prefer absolute timestamps rather than relative.	2
Repetition within the Discussions tab	Connected with the above. When multiple changes are made in a short timeframe to a diff, they are all given the same relative timestamp. Without togging the discussion, it is impossible to distinguish between them.	3
Issue Number	No users used the widget or Merge Status tab to identify the MR's related issue number, they found it in the MR description. Users commented that they would like the issue number to be visible at all times, especially if the description is truncated.	3	This may not be a problem to users if we chose not to implement the description expansion.
Emojis	Did not understand the significance of the emojis. One user thought they indicated that the MR had been reviewed by 5 people (as opposed to 5 'thumbs-up').	3
Task List	Users were unaware that they could create a task list in an issue description.	2
Discussion Tab	Users felt that the word 'Discussion' didn't accurately describe the content within the Discussions tab. One user stated that it is a "thread of activity". He also pointed out that we use the word 'Comments' within the Changes tab, subsequently he felt there was some inconsistency in how we've chosen to label similar content. A second user stated that everyone uses "comments" on the web, rather than the word "Discussion".	2	Users may have been influenced by how the task question was phrased i.e. "Who last commented..?"
Merge Status tab as default	Multiple users who are used to GitLab's existing experience, accidentally visited the Merge Status tab when trying to view Discussions.	3	This obviously slows down users and causes some frustration. It takes time for users to adapt their mental modes.
Iconography on Merge Status tab and within widget	One user felt we were overusing the green checkmark (passed) and red cross (fail) icons. He comments that they were originally used to specify the status of a Pipeline build, but they seem to be creeping more into the interface. For example, 'No changes to code quality' has a green tick but it isn't a job.	1
Pipelines	Within one stage, there could be more than one build. If 2/3 of the builds within a stage fail, the user clicks Retry. However, this redirects them to the 'progress page'. Subsequently the user has to go back, refresh the page, re-click the failed stage icon followed by Retry. The user doesn't want to be redirected.	1	I'm not sure if there's a reason why we redirect users?

Merge Request Page - Results

Researcher Todos

Script

Prototypes

Screening criteria for users

Videos

Summary of findings

Analysis

Which prototype did users prefer?

Completion of key tasks

Usability Issues

Designs

Child items ...

Activity

Script assessment points:

Findings 1

Findings 2

Admin message

Admin message

Merge Request Page - Results

Researcher Todos

Script

Prototypes

Screening criteria for users

Videos

Summary of findings

Analysis

Which prototype did users prefer?

Completion of key tasks

Usability Issues

Relates to

Activity

Script assessment points:

Findings 1

Findings 2