[Meta] Making GitLab Real Time

With the implementation of #5983 (closed) (uses Vue.js) by @selfup GitLab is starting to better reflect the current situation. That issue will take care of updating the pipeline list and pipeline graph's in real time. This is a meta issue to organize our efforts in making sure GitLab is consistently real time throughout the application, making sure we don't confuse users whether a view updates itself yes or no.

The result of this effort would create a living environment which consistently informs our users correctly. We will have to gauge wether some things are worth making real time and how this will effect UX. GitLab is like no other and has this incredible potential, let's make that a reality.

Technique/Implementation

I have heard rumors about pub sub and other techniques (Periodic polling with increasing delay when there's nothing new) in order for us to make this possible. Has there been decided on a stable implementation in order for this all to be possible which is future proof as well? @jschatz1

If decided let's update this issue body with information on this.

Handling cached views

I have seen various issues about this already. If a page/element is considered real time, will it make sure it reflects the current situation accurately even if the user has decided to go back to previous page, loading a cached version of it?

Important to get this down correctly and consistently throughout the application! Put information here

UX and Micro animations

Real time can have a lot of impact on the UX side of things. We will have to see if we provide enough feedback to the user. And if we don't do things which may negatively impact a users ability to act upon information (moving things off screen to quickly as new updates are rendered on top of them being one example).

Feedback can come in various formats, one of those being micro animations. Letting an application feel alive will be enhanced by this and can in most cases probably done by css alone. Micro animations should in no way draw too much attention, merely inform (like all other information). On top of that, performance should never be impacted by this.

Important to get this down correctly and consistently throughout the application! Create a guide upon this topic here

General rule: Old element should just disappear with no animation, and the new elements should have a quick fade in. Nothing more. reference

Plan

Some pages will have a higher priority then others to be real time, this issue body should be adjusted to reflect that.

PS: I was busy fleshing this out completely with elements and example images, but it takes too much time, if necessary I will do so.

The idea is that we keep oversight in this issue and create issues from this list to be acted upon.

Personal Navigation

• GitLab Sidebar

  • Your Todo Count
  • Your Issue Count
  • Your Merge Request Count

• /dashboard/projects (low priority)

  • Project lists
    • New items
    • Name
    • Description (low priority)
    • Avatar (low priority)
    • Pipeline Status icon
    • Star count
    • Tab count badges (not yet there, issue: #24423 (moved))

• /dashboard/todos

  • Todo lists
    • New items
    • Comment author time
    • Comment description
    • Tab Count badges

• /dashboard/activity

  • Activity list
    • New items
    • Activity Author time
    • Activity Content

• /dashboard/groups (low priority)

  • Groups list
    • New items
    • Name
    • Description
    • Role
    • Project count
    • Member count
    • Settings icon
    • New items
    • Visibility

• /dashboard/milestones

  • Milestones list
    • New items
    • Name
    • Content
    • Progress bar (you could watch it grow!)
    • Tab count badges (not yet there, issue: #24421 (closed))

• /dashboard/issues

  • Issue list
    • Tab count badges
    • New items
    • Name
    • avatar
    • Confidentiality
    • Creation time
    • update time
    • comment count
    • thumbs up count
    • assignee avatar
    • milestone
    • tags
    • status

• /dashboard/merge_requests

  • Merge Request list
    • Tab count badges
    • New items
    • Name
    • avatar
    • Creation time
    • update time
    • comment count
    • thumbs up count
    • assignee avatar
    • milestone
    • tags
    • status

• /dashboard/snippets (low priority)

  • Snippets list
    • Tab count badges
    • visibility
    • name
    • filename
    • author time
    • avatar

• /help (probably not applicable)

• /profile (probably not applicable)

Project wide navigation

• Project wide navigation elements
  • Project Name (not important)
  • Issue count
  • Merge Request count
  • Your Todo Count (Important)
  • Your avatar image (not important)

Project Views

• /projectname

  • Readme.md
  • Last commit
  • Commit Pipeline Status
  • Stat counters
  • Name
    • ssh/https url
  • Description
  • Stars
  • Forks

• /projectname/activity

  • New items
  • Activity Author time
  • Activity Content

• /projectname/tree/master

  • last commit
  • files list - all elements

• /projectname/commits/master

  • commit list

• /projectname/network/master

  • graph (you would be able to see it grow real time.. zooming out would be incredible here)

• /projectname/branches

  • branches list

• /projectname/tags

• tags list

• /projectname/path-locks

  • locked files list

• /projectname/pipelines

  • #5983 (closed) Pipeline List - All elements

• pipelines/ID

  • #5983 (closed) Pipeline Graph
  • Build list
  • Meta information, such as pipeline status, author time

• /projectname/builds

  • Build lists

• /projectname/environments

  • Environment lists

• /projectname/cycle_analytics

  • Cycle analytics

• /projectname/Graphs/master

  • contributors graph
  • commits graph
  • languages used graph
  • continuous integration graph

• /projectname/issues

  • Issues list

• /projectname/boards

  • Issues board #24478 (moved)

• /projectname/Merge Requests

  • Merge Request list

• /projectname/Labels

  • labels list

• projectname/Milestones

  • milestones list

• /projectname/Snippets

  • snippets list

Detail views

[ ] To be added.

---

The reach of this issue is enormous (although we will have to see exactly which scope we want this to be... Probable moonshots issue.

cc: @markpundsack I think ~CI benefits the most from this and should therefore be our prime target in this endeavour

Added ~19173 label

Mentioned in issue #24478 (moved)

making issue boards real time: https://gitlab.com/gitlab-org/gitlab-ce/issues/24478

Glad to hear someone else is thinking about this @dimitrieh! It's possible that the talk about polling and pubsub you mention are related to the work I've been doing on the frontend for timetracking. From the infrastructure I created for that, I'm hoping to be able to have all Issuable (MR/Issue) sidebar components being updated in 'real time' (or at least as often as we want) in this coming release.

It has proven to be a pretty sticky task to get working for the issuable sidebar components, mostly because they're used in so many places. I suspect it will be easier with one-off pages and/or components.

I would love to help with this however I can!

@dimitrieh
Maybe a table view would be better than that lists of lists in the issue body?

@blackst0ne great idea :P although I hate editing tables with markdown

let's first get more idea's in here on how to organize this effort in terms of which issues/pages/elements to tackle. Then adjust the main issue body accordingly

@brycepj great to hear! We really should get @jschatz1 attention to this issue as it seems various people are putting work in making gitlab real time already.. to make sure we are doing this the right way and not use different kinds of techniques resulting in double work in the end!

related: https://gitlab.com/gitlab-org/gitlab-ce/issues/23276 as for indications of real time updating

Mentioned in issue #24643 (closed)

Mentioned in issue #23276 (closed)

Showing CI build status and having it be up to date without clicking refresh would be huge.

I often demonstrate the product to people and hit refresh a lot on the CI pages so I can see a build go from Pending to In Progress, and then either to complete and happy (green) or complete and sad (red).

@warren.postma this will be partly covered by https://gitlab.com/gitlab-org/gitlab-ce/issues/5983

Todo badge counter to be realtime across pages: https://gitlab.com/gitlab-org/gitlab-ce/issues/24160

@victorwu perhaps you can use this issue to manage/populate it with issues making gitlab realtime/always up to date across screens.

@dimitrieh + @awhildy : Do you have any particular ideas on where auto-refresh should start first? @jschatz1 says we are in a good position to start pushing this.

1. My thoughts are that issues would be a good area since that gets a lot of visibility. In particular issue description could be a good starting place so that we can at least dog-food it when we are all starting at the same issue description. That's the safe strategy I would say?

2. Regarding the UI, do we have any consistent designs? For example, if the issue description suddenly refreshes by itself, should their be a brief spinner appearing, even if to insert a fake time delay just for UI reasons?

@jschatz1 : Is this related to https://gitlab.com/gitlab-org/gitlab-ce/issues/23207 in any way technically? Any concerns there?

@victorwu: See my comment here: https://gitlab.com/gitlab-org/gitlab-ce/issues/23276#note_18645550

Basically, I think we should have a nice quick subtle transition (from my experience, the smoothest is cutting out the old content, and then a quick fade in of the new content to avoid cross-fade). I don't think we need to add any sort of delay or spinner, etc. We will build trust with users that our content is live by having it live. Broadcasting it too much distracts a user from their current flow/focus. There might be a few key moments where it is nice to broadcast an update louder, but I'd rather start from a baseline of smooth and quick, keeping things quiet and focus so when we want to draw attention to something, it is much easier to and the impact is larger.

Makes sense to start with issues . Personally, I can't wait to have everything real time, but starting with an area that we personally use a lot makes sense so we can work out all of the kinks.

Thanks @awhildy for the design. Let's run with that. (And @dimitrieh already described it in the description. Thanks)

How about we do everything in the issue view: title, description, status/buttons and then issue sidebar metadata, and maybe clean up comments if necessary. That way we would go deep and get feedback on a single view over time. And our users would sooner get a consistent experience seeing everything in at least one page is autorefreshing, rather than sporadic elements throughout the app.

I'll go ahead and add a few issues and update the description here to link to them, and also make sure to link to your fade-in design described in already a few places. Let me know if you want to prioritize in any other order.

@victorwu @awhildy Just a note on the issue description changing real time. I think this is an edge case, which needs a slightly different way of handling things. Instead of it showing a spinner for a moment and it being updated after a delay.. let's have a white fade on it with a message: "This Issue/MR Description has been updated, click here to refresh". This as people may be referencing stuff from this issue body and its editable by anyone. Do you agree?

Mentioned in issue #25049 (closed)

Mentioned in issue #23225 (closed)

Mentioned in issue #2512 (moved)

@dimitrieh: Not positive I agree. I feel like the whole point of real time is that it is updating real time. Having to click to refresh I think makes things not feel great 100% of the time for the 5% of the time (or less) that someone is deleting the thing you are actively referencing. I'd rather explore ways to see the previous versions of the description. For example, Google Docs always shows you the latest document, but you can click on View changes to see a history of changes.

We shouldn't necessarily show a spinner over it while it is updating. I think we should just show the change with a fade in when we get the update, and a 'last updated' message so you know when it was last updated.

@dimitrieh About the pub/sub technique, websockets would be a possibility. In Rails < 5, we would rely on the websocket-rails gem which I found a bit cubersome to use; in Rails 5, websockets are integrated via Action Cable

There is actually an extensive discussion on websockets in #5983 (closed). They don't scale well and ActionCable cannot be used on a non multi-threaded web server like Unicorn

@awhildy mmh I see. Perhaps its assumping something needs to be fixed.. thinking to much ahead. I would be okay to just have it update in the way as you say.. so we can later always iterate on it if necessary! This problem as a whole would of course go away if we implement real time editing, like google docs in which you can see somebody else write: https://gitlab.com/gitlab-org/gitlab-ce/issues/4199

mentioned in issue #19745 (moved)

@brycepj probably nice to reference you in this issue, if you start working again on the real-time updating of the sidebars of the issue and mr views.

cc: @victorwu (as I mentioned you in all the other related issues of this subject)

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8108

mentioned in issue #25417 (closed)

mentioned in issue #23892 (closed)

mentioned in merge request !8357 (closed)

#26396 (closed) Is going to be the brainstorm for how we can implement realtime the same way for every feature. This way we can ensure it scales.

Just a mention here to the ~Frontend people from the UX webcast has there been looked at http://socket.io/ ?

asked it in the chat too ;) https://gitlab.slack.com/archives/frontend/p1484853605006641

edit: apparently it involved websockets cc: @lbennett

Yes this is something I would love (I have used it before as well) but will not be easy to add to our stack omnibus.

It only runs on Node, and we currently do not use node for any runtime operations.

Once we figure out the new plan that will be used to make GitLab realtime in the very near future, then we can iterate over that.

I think that first we see how well the new plan will work.

Currently the plan is to not integrate Node into our Rails/Ruby and Golang stack.

Golang can do websockets as well, so we can potentially run websockets for true pub/sub in the future instead of short or long polling.

Thanks for looking around!

mentioned in issue #26664 (moved)

mentioned in issue #27452 (closed)

@jschatz1 do you know if making the todo list real time is anywhere being considered?

@dimitrieh - I haven't seen that anywhere. Maybe create an issue, and we can close it if we find a duplicate?

mentioned in issue #27825 (moved)

Created: https://gitlab.com/gitlab-org/gitlab-ce/issues/27825

mentioned in issue #28041 (closed)

making the job page real time with pipeline timing information: https://gitlab.com/gitlab-org/gitlab-ce/issues/8100

mentioned in issue #29480 (moved)

mentioned in merge request !10028 (merged)

mentioned in issue gitlab-com/organization#69

@dimitrieh we should def do that.

mentioned in issue #31227 (moved)

mentioned in issue #29567 (moved)

closed

Closing for https://gitlab.com/gitlab-org/gitlab-ce/issues/31227#note_27875319