Update pipelines table in real-time

Absolutely. Adding @grzesiek @ayufan for input from CI side.

Added ~19173 label

Completely agree that builds page should be updated realtime. I had some thought about using ActionCable for this (there is relevant issue somewhere), but any mechanism, that will make it real-time, will be good.

@jschatz1 we already have yellow colour to highlight changes in the right sidebar, so it can be reused in this issue (if it's too bright, we can try #fff7d8).

The only question is for how long is it needed to highlight updates for builds - till the page refresh for sure. But if a user stays on the page for longer time, which is a real case for managers- we need a time gap of 30 min maybe before stop highlighting. What do you think?

@grzesiek Maybe we can just periodically refresh the page if the builds changed, ci_builds.updated_at?

@ayufan This probably would be most simple solution, but I'm not sure if it is most refined one.

@grzesiek We can do ajax request to refresh only builds part.

Does this cover merge request build status, too?

Added ~113220 label

Added feature proposal label

Absolutely

Nice is also quite interesting and useful feature, I think. Maybe it is worth scheduling, what do you think @ayufan, @JobV?

@grzesiek It will be hard to implement until we don't start using Websockets. As for now we can only periodically refresh the views only or do some ajax calls.

@ayufan We can wait for Rails 5 with ActionCable, start using ActionCable earlier or go with simple refresh first.

@grzesiek @ayufan true real time we need websockets but quasi real time we can poll for a simpler, easier, boring to implement solution. Just options, maybe should wait for websockets.

@jschatz1 The websockets are not gonna happen soon enough, we should start with simple refresh by using ajax or json update. We are doing that for refreshing build log.

@grzesiek Feel free to schedule it and do it by yourself :)

mentioned in issue #13559 (closed)

I'm with @ayufan. Start simple. Upgrade to Rails 5 (sockets) will be a numbers of months after its release.

I think for right now this is a frontend issue until sockets come along. I'm adding a frontend tag.

Added frontend label

mentioned in issue #14657 (closed)

mentioned in issue #14717 (closed)

Should we have an issue or label to track all the issues that can't be resolved until ActionCable is added?

@connorshea I've created a websockets label for any issue that will use websockets once they become available.

mentioned in issue #15598 (closed)

mentioned in issue #17686 (closed)

Milestone changed to %Backlog

Changed title: Builds should update in real-time → Pipelines/builds should update in real-time

mentioned in issue #19088 (closed)

@jschatz1 I believe we spoke about not using sockets for this?

@JobV Really? That's surprising, as this seems ideal for sockets.

@rspeicher I don't know what I'm talking about. I heard someone say that FB doesn't use sockets, just polling, because it's easier / more boring to scale.

I mean, you could make an argument for using server sent events (SSE) rather than websockets. But polling? I find that hard to believe. :)

@rspeicher @markpundsack according to @northrup (in a discussion we had in Austin) sockets are very very hard to scale. He said even subsecond polling (as FB does) is much more scaleable.

Fair 'nough.

I have to wonder if that's for particular languages such as PHP (and probably ruby) that are kinda heavy when handling multiple connections, as opposed to Go, Node, or even Java that can scale better.

mentioned in issue #20051 (closed)

Added websockets label

mentioned in issue #17759 (closed)

Changed title: Pipelines/builds should update in real-time → Pipelines/builds should update in real-time or at least refresh

From #20051 (closed):

When we fetch data we could store the updated_at information that is present in Pipelines and Builds. This would allow us to simply check when anything got updated, thus would be a good indicator to refresh these views.

This is pretty much a solved problem, so shouldn't be hard to do. It's just a matter of deciding to do it. :)

We have options ranging from:

Add <meta http-equiv="refresh" content="5"> (Please don't really do this)
Refresh while respecting turbolinks
Use ajax to poll for changes and fetch pre-rendered HTML partials to replace existing HTML elements
Use ajax to poll for changes and fetch JSON, render replacements using Javascript
TurboLinks 3.0 partials?
Use Socket.io
Use Pusher service
Use ActionCable after upgrade to Rails 5
Use ActionCable now

Personally, I'm not worried about WebSockets not being scalable. We're not Facebook.

I'm not worried about needing to flash the tab or updated section either. In my experience, if you're watching for a build to update, you're going to notice the icon change, or a new pipeline appearing at the top of the list and everything shifting down. I've fought with these patterns before, from considering have a "New data" alert somewhere, so someone has to click to see the new data. It's not worth it. That stuff is needed for Facebook where you're pushing data at the top, while the user is reading towards the bottom. When you're watching the top, you expect realtime to just happen.

I'd love to see us try ActionCable without waiting for Rails 5, but I'm pretty sure it's going to require Puma instead of Unicorn. You just can't keep a process open for every websocket; you need threads.

What do we use for pushing new commends to the Discussion feed of Issues? How about we start with doing the exact same thing?

/cc @annabeldunstone @ayufan

What do we use for pushing new commends to the Discussion feed of Issues? How about we start with doing the exact same thing?

@markpundsack Periodic polling with increasing delay when there's nothing new. So we poll for a note every 15 seconds, if nothing's changed we fall back to 30, 60, then 120 at the maximum. Then it resets if something happens on a poll. It's not great but it works.

That sounds like a great start!

mentioned in commit maxraab/gitlab-ce@38397dbe

mentioned in issue #20902 (closed)

mentioned in issue #17589 (closed)

My recent thoughts about it on Slack: https://gitlab.slack.com/archives/development/p1471247124002612

I'm not saying that websockets are the right solution for everything, I merely say, that if we really want to use websockets then actioncable is something worth trying. The problem is that ActionCable does not seem quite ready yet. I've been using this in production already with Rails around 4.0, but there is still one blocker to do this right - no good testing technique yet - https://github.com/rails/rails/pull/23211, and related - https://github.com/rspec/rspec-rails/issues/1606

And I'm not sure we can properly use any pub/sub/two-way communication thing unless we can work out a good testing approach.

For now, as far as I know, ActionCable looks like the most promising real-time like communication channel that will be able to integrate nicely with our Rspec feature examples suite, but this is not ready yet. With other tools we may need to follow end-to-end testing route, which will most likely mean that we won't test things good enough.

The bottom line is that we are in big need of end to end tests, so we started working on https://gitlab.com/gitlab-org/gitlab-qa. But this test suite is, in my opinion, not the place where specs for real-time code belong to. We need good testing approach before adding Faye/ActionCable/AnythingElse.

/cc @rspeicher @DouweM

mentioned in issue #14852 (closed)

mentioned in issue #21372 (closed)

mentioned in issue #21277 (closed)

mentioned in issue #21487 (closed)

mentioned in issue #21746 (moved)

Builds list and build status page are like the only pages that have to be dynamic in the whole GitLab. :)

@jschatz1 It's about time we picked this up. :) Do you think we could plan for it in 8.13? Specifically what I'm asking for is to do updates the same way as done elsewhere in GitLab, as described in https://gitlab.com/gitlab-org/gitlab-ce/issues/5983#note_13212307. I think websockets has to be out of scope for now. This might be a bit too much to ask @annabeldunstone to pick up. Anyone else on your team have experience with doing the existing update mechanism that could help extend that to Pipelines?

Rails 5.0, with action cable, was released in June, so the tools are there. What's blocking this from happening?

/subscribe

@frederikcreemers ActionCable requires web server that is multi-threaded, like Puma. We currently use Unicorn while we are still working on memory leaks that prevent us from using multi-threaded server.

@grzesiek, are there any issues about preventing memory leaks, just to get subscribed to them?

@frederikcreemers See #14286 (moved) and !5555 (closed), there's still plenty of work to do in order to get GitLab onto Rails 5. I've spent a lot of time the last few months getting us ready to upgrade, and I can say it's no easy task :) There are still a few gems holding us back, and even then plenty of test failures on the MR.

I believe our current intent is not to use ActionCable, or maybe even not use WebSockets. There are infrastructure concerns with WebSockets being too resource-intensive, among other things.

@grzesiek are you sure? Last I heard that was no longer a requirement.

@connorshea I don't really know if we want to use ActionCable or not, but while we can still have main web app on Unicorn, ActionCable worker needs to use multi-threaded server. Does it answer your question?

Yes

@frederikcreemers @northrup has a good opinion on this. When we talked last he said messaging with pubsub is much more realistic and scaleable.

The problem with WebSockets is that when your application reaches any kind of scale or size load balancing for connections that stay open to the server becomes a problem and does not scale. I would posit that event status updates in GitLab and CI Project runs, while very important, don't require the overhead of maintaining a websocket connection, these are items that can be polled on a frequent basis by the client. Likewise, I think that the submission of data from the CI runner back to GitLab should be an update thrown on a bus to be consumed by a worker(s) processing the information, not realtime API calls yielding touch-backs all the way to the database.

@northrup

Looking in my browser it appears that Facebook is using long polling :)

@northrup

Connection is up for 50s and is flushed from time to time to deliver new data to browser.

Mentioned in issue #21698 (closed)

@annabeldunstone @lbennett @filipa This came up as part of the Idea to Production demo, but has been sorely needed for a while. Any chance of picking this up for 8.13?

Milestone changed to %8.13

Changed title: Pipelines/builds should update in real-time or at least refresh → Update pipelines in real-time or at least refresh

Added ~122770 label

Removed websockets label

Updated description with proposal to do exactly what we do elsewhere. Also removed:

Bonus: If updated in realtime and done properly you could get the flashy chrome tab automatically. Like in Gmail when you get a new email the tab flashes to show you new things happened. See this

@ayufan Facebook use pub/sub architecture, and specifically when it comes to needed to update the users in 'realtime', like the messenger app in the browser (and on the phone), they use MQTT.

The simplest what we can do now:

When generating a page fetch latest updated_at of pipeline,
Ask every some time (famous active pulling) to see if updated_at did change,
Refresh page when needed.

The next step would be:

Use Redis long long polling pub/sub channel to make backend to force refresh on client side,
The backend mechanics will be pretty much the same.

Also, when there's a time indication, e.g. 15 seconds ago, expose it to HTML as <span data-original-time="2016-08-01 12:12:13">15 seconds ago</span> and update the counter in JavaScript.

Reassigned to @filipa

Assignee removed

Is the description proposal updated and the single source of truth? I think I will be able to start working on this today/monday morning 😄

//cc @jschatz1 @ayufan @markpundsack

Should we really just be refreshing the page? 😞 I understand it simplifies a lot but that really messes with the UX if it you're half way through reading/interacting and the page gets taken away from you?

How often on average for gitlab-ce does updated_at change? Using knapsack should mean that we have a space in the middle where updated_at doesn't update very often, but at the start and at the end of the pipeline process it should update more frequently?

Also forcing the browser to refresh without input from the user seems kind of silly because users already have the ability to refresh whenever they feel they need to?

On the other hand, it could be one of those things that only looks like a problem on paper but in reality isn't a sacrifice at all.

Mentioned in issue #23069 (moved)

I agree here with @lbennett

Refreshing the page on something else updating is imo bad UX, also I think there is potential for a lot of refreshing.. nobody would be happy with that.

Mentioned in issue #22190 (closed)

@lbennett Would you propose something like we do with the ci_status? So polling and let JS update the elements on the page?

@zj @fatihacet I created a subtle animation for the running icon to give feedback to the user that there is actually something happening on the page.. that it will refresh/update once a "running" build has succeeded or failed.

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6800 (the implementation is an abomination but shows of what I want it to! ;) see:

We can do the animation as slowly as we want, but I think this is oke.. We have to see how it looks in real conditions.. as the pipelines in the "GDK" doesn't represent a real situation.

@markpundsack @awhildy do you agree that this adds value? Its something that was on my mind, thought I'd create a way of showing this off, let me know what you think.

My only reservations to this is that it should be implemented well and not be distracting.

It looks cool, but is a bit risky. What does it look like when there are 40 of them? Are people going to think that when it finishes a cycle of animation that that means that the job is actually finished?

@markpundsack

What does it look like when there are 40 of them?

This is a thing that I said we have to look to see how that is... Just took a look at pipelines tab and the top 10 can be running or so. A possibility that makes this less of a thing, is asynchronous animations (at least I think so) and perhaps seeing what timing works best.

Are people going to think that when it finishes a cycle of animation that that means that the job is actually finished?

I don't think so if its a continuous animation.. it doesn't stop and after 1 cycle.. people know for sure that its just an animation. Although there is a possibility here for a "progress indicator" :) although that complicates this and isn't the point.. rather an iteration on top of this.

@zj

Would you propose something like we do with the ci_status?

Exactly! It will be a lot more involved but at least we wouldn't be taking away from the UX.

@dimitrieh When I first saw this I went "COOL! DETERMINATE PROGRESS ANIMATION!" but then continuing to read I find out that this is in fact indeterminate and will continue to spin until complete. IMO, firstly this doesn't seem to be the best animation for indeterminate progress as it denotes some degree of progress. Secondly, having that icon run continuously gives no more information to the user as does as a static icon, so feels kind of unneeded.

However, if this was animating to the actual % complete the build currently is, that would be epic.

I can't think of a way to get a % completion on the build, unless instrumented in some way by the user, or very crudely, by counting the number of commands in the CI configuration for that stage, and seeing how many have been executed.

@frederikcreemers Indeed,

by counting the number of commands in the CI configuration for that stage

This is what I was thinking, but as you've stated, it's pretty crude.

@lbennett ha nice

Secondly, having that icon run continuously gives no more information to the user as does as a static icon, so feels kind of unneeded.

The Initial idea why this is needed it the following: "to give feedback to the user that there is actually something happening on the page.. that it will refresh/update once a "running" build has succeeded or failed."

this doesn't seem to be the best animation for indeterminate progress as it denotes some degree of progress

I agree to some degree 😄 but this is closest to the icon we already have ;).

However, if this was animating to the actual % complete the build currently is, that would be epic.

This perhaps solves the the other quote completely, although it makes it a little bit more difficult 🎱

sidenote here: this icon can be quite small.. thus you can only see something like.. oh its halfway done or its at around 30%.

Another thing to note is: a lot of the running builds are stuck at the beginning cloning the repo in order to do the tests on them afterwards. We should always have progress to some degree... something like 15% or 35% to let the icon be visible.. otherwise it would just be an empty circle :O

I say skip the animation.

I agree with @markpundsack here. Let's not add cool things that confuse the experience more. The proposed animation looks like the job will be finished once the circle is complete, which is not necessarily the case. Making the animation be based on the actual % of completion seems needlessly complex. If we were to show an animation, it should be a loading indicator that represents indeterminate progress as @lbennett mentioned.

@tauriedavis I can relate, still its a cool idea ^^ https://gitlab.com/gitlab-org/gitlab-ce/issues/22192

If we were to show an animation, it should be a loading indicator that represents indeterminate progress as @lbennett mentioned.

This would mean changing the icon.... and is kind of related to: https://gitlab.com/gitlab-org/gitlab-ce/issues/20084

..damn the gear icon would fit this really well also

edit: @markpundsack @tauriedavis I commented here: https://gitlab.com/gitlab-org/gitlab-ce/issues/22628#note_16811696 based on this comment, please check it out ;)

Mentioned in issue #22628 (closed)

@dimitrieh Is there a state where the icon would be the running icon but not animated? If not then maybe it's best to keep it static? I might just be boring though 🏃

@lbennett not to my knowledge...

And no problem.. I am probably paying to much attention to this, but rather be sure and exploring every possibility than the other way around ;)

@tauriedavis thought I'd share some alternatives to the cog icon ;) in the sense of

it should be a loading indicator that represents indeterminate progress

speed can be adjusted

Live Demo

Thanks @dimitrieh, maybe for a next step it would be good to see one of these in place with multiple of them on the page to make sure it is not overwhelming to look at many moving animations on one screen?

Animations are messing with my head tbh, if something is running, its cool to see a blue icon. But other than that, it doesn't require my attention, I just have to wait. So I'd like a static asset I think. Just my 2c

Mentioned in issue #23276 (closed)

@zj Alright let's see, the main point of having an animation is to give feedback to the user that the actual page is live! ;) I created an issue for this to take this on in a later release ;)

https://gitlab.com/gitlab-org/gitlab-ce/issues/23276

I fully agree this should be a hint and not be distracting at all! The animations currently are pretty big and right in your face.. this should not be the case at all

Mentioned in merge request !6800 (closed)

@lbennett @annabeldunstone @filipa @jschatz1 Is anyone working on this issue for 8.13? (I mean the original goal of getting realtime updates, not the sidetracked discussion of icons :) )

@markpundsack I actually started working on this but then @ayufan decided to take it easy for 8.13.

@fatihacet There will still be probably a few MR that will require your review, so I would prefer them to be merged :)

Milestone changed to %8.14

Mentioned in issue #22689 (closed)

Mentioned in issue #23492 (closed)

Added ~307502 label

@jschatz1 Is it reasonable to expect this to ship in 8.14? Who's working on it?

This seems that it will require some amount of Backend help to deliver the data. Not sure about exact amount required.

Reassigned to @selfup

@selfup I see you're assigned to this -- might be a great candidate for SubbableResource and/or SmartInterval (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6858#note_17487301). Let me know, and I can keep you posted on their progress. If you don't think they'll have what you need, let me know that too.... I'm currently trying to make them more reusable.

Hello, i apologize to interupt yours intern discusion. But have you heard about nchan ? It is a module of nginx handling pubsub server using a lot of tranport techniques (websockets, long-polling, SSE, interval polling, and more) you spoken about in this discussion and is scalable (experimentaly) via redis. Isn`t it candidate to backend worker?

Mentioned in merge request !7155 (closed)

@brycepj That looks awesome! I am going to try to re-write this in Vue.js instead and use 0 jquery. So we shall see if I need to use it :)

@mardukecz

Nchan looks interesting, could you provide more details about it? Like who uses it, are there open code bases that use it? (Maybe even using rails?)

Are you the maintainer btw? Also, we support Apache officially. The name and docs suggest no support for Apache. Is that right?

Mentioned in issue #14852 (closed)

@zj I met nchan when i create our private CI/CD Board. On start it loads a state of project via API and then listen to websocket channel of nchan which consume translated webhooks. I am only Java senior programmer, fan of Gitlab and promoter of it in our small company.

Nchan is module of nginx, but it can be configured as a standalone server, which enable using in combination with other web server. Usage is easy, set endpoints to publisher and subscribers. Put new messages (typicaly POST) to publisher endpoint and "connect" client to subcribers endpoint and all nchan sends them. I must change only one thing, default conf is that on connect (neither type) sends all memorizes messages, this can be configred to whole history (stored) last X message, none or skip next X messages.

I dont use all features but, i choose it for this features:

publish message to channel (POST for HTTP or websocket, can use both), multiple endpoints to same channel possible
same url or set of urls to subscribe channel, each can specify multiple types of serving technique
storing messages (disk, memory or redis)
debugging meta channels
support static channels and dynamic channels which is optimal for pipeline content refresh, issue/mr discussion update etc. https://nchan.slact.net/#the-channel-id
Channel Multiplexing (limited to 255 channels) - https://nchan.slact.net/#channel-multiplexing
securing channel with authetication - https://nchan.slact.net/details#securing-channels
possible connection more nchan servers via redis (cluster)
is a module to our webserver and there no need to write own websocket server

And now cons:

redis is still experimental, we dont test it
is young software, we use debian and our sysadmin says that he must download some testing packages to download nchan, not much https://packages.debian.org/stretch/libnginx-mod-nchan, i dont how it is going in other distributions
i dont found any org that use it, we use it on small task without problem

Is it enough?

As I don't know nchan I can't say much about it but I'd rather go with GraphQL, see https://gitlab.com/gitlab-org/gitlab-ce/issues/22753. It allows API calls and also offers pub sub without the need to maintain two separate technologies (REST + PubSub solution). This makes frontend development a breeze as you don't have to map manually the data coming from REST together with some live data stream coming from e.g. nchan inside of the web client. Also Facebook (well they've actually invented it), GitHub and other big players are using it already so it definitely has potential to become a Web Standard.

Please give it some throughout thought :)

Mentioned in issue #22753 (closed)

@subesokun This would still need polling right? I don't understand how GQL has anything to do with the issue at hand? Its a query language/structure not a technology for doing anything live like websockets or http/2 could provide?

Please explain where I am wrong here.

@zj Exactly, GraphQL is just a query language and you'd need a client and server which implements it (see below). With GraphQL you can specify subscription queries which are event based in order to receive pushes from the server. This blog post might help to understand the idea behind it http://graphql.org/blog/subscriptions-in-graphql-and-relay/

A potential technology stack you could use which implements GraphQL is the Apollo Stack. It provides both, a client and a server, and it seems to be quite trivial to integrate the Apollo client into Vue.js. Actually there is already a library for it called vue-apollo (example) but the Vue.js plugin for Apollo looks very tiny so you could implement also your own one if you like. Furthermore the Apollo client supports GraphQL subscriptions.

Regarding the backend, as GitLab is using Ruby I can't say how easy/difficult it will be to integrate the Apollo server there or if it's even possible. The input of a GitLab backend expert would be very helpful here. Frontend wise I think it's quite straight forward to implement the Apollo client as GitLab is using now Vue.js.

Nchan looks interesting, could you provide more details about it?

Like who uses it, are there open code bases that use it? (Maybe even using rails?)

Are you the maintainer?

Also, we support Apache officially. The name and docs suggest no support for Apache. Is that right?

@zj I am not maintener. I could not find any product which use it. As i wrote 4 days ago, in cons project is young. On official site is written The present release is in the testing phase Nchan is nginx module, so there are two ways to use it. As a module, or standalone

@mardukecz This was a repost, as now mail_room was processing again. Sorry that you had to answer the same questions twice 😞

It was confusing. No problem 😃

Mentioned in merge request !7295 (closed)

Mentioned in issue #24429 (closed)

Mentioned in issue #23712 (moved)

Milestone changed to %8.15

Added Deliverable label

Removed ~307502 label

Mentioned in issue #24643 (closed)

Sorry, just wanted to make sure if this was still on track for 8.15?

@jason-riddle I think it will go in 8.16.

changed milestone to %8.16

mentioned in issue #25803 (closed)

mentioned in issue #25419 (closed)

mentioned in merge request !7196 (merged)

mentioned in issue #26396 (closed)

@jschatz1 @selfup Should we close this issue now that !7196 (merged) has been merged?

@rymai As far as I know pipelines page does not upgrade in realtime yet!

Currently the realtime aspects are blocked by the realtime diff updates on the backend. This will be handled in a different merge request once this one is merged.

mentioned in issue #26919 (closed)

As @grzesiek said we keep this issue open. Infra/Backend is working on a realtime solution (cache hit/miss). Then this page will be plug and play!

mentioned in issue #26259 (closed)

changed milestone to %Backlog

Not sure this is the right place but;

I'm pretty sure that before when you were looking at a build; the moment the build/stage finished you were redirected to the first build of the next stage.

What happened to this feature? Should I make another issue?

@oomathias 🤔 I don't remember that but I haven't been around for that long 🙊 😸 Can you please create another issue? I am afraid it will get lost in this one :)

I'm pretty sure that before when you were looking at a build; the moment the build/stage finished you were redirected to the first build of the next stage.

@oomathias I agree, this was working like that, didn't check if that changed, but if you're confident it did, please create an issue.

@filipa @rymai Thanks! Issue created https://gitlab.com/gitlab-org/gitlab-ce/issues/27218

mentioned in merge request !8878 (closed)

added pipeline label

mentioned in issue #8100 (moved)

removed assignee

mentioned in issue #29131 (moved)

changed milestone to %9.1

@ayufan @jschatz1 just to be sure, we want a simple backoff pooling here, correct?

I just have one question:

should this pooling be always on, or should we disable it if the user opens another tab in the browser?

@filipa

Yes, simple pooling. I would love to hide that behind a feature flag if possible and see how it affects GitLab.com and merged Etag solution. I'm not sure about Backoff, I would prefer to avoid that. We need some serious backend to make this happen. I still think that we need differential updates delivered to the client as we started doing with @selfup in the past.

How about this:

Can we just adapt @selfup method with naive polling first behind a feature flag? We always do full fetch,
We add an Etag approach,
We consider if we can make it more optimised (by stripping data?) out of payload doing incremental updates as discussed in the past.

@ayufan The description suggest a backoff pooling:

Periodic polling with increasing delay when there's nothing new. So we poll for a note every 15 seconds, if nothing's changed we fall back to 30, 60, then 120 at the maximum.

That's why I asked.

Can we just adapt @selfup method with naive polling first behind a feature flag? We always do full fetch,

I don't know any code on gitlab that does this. @selfup what was already done, where can I find it and why did we not merge it? (Just trying to get the full picture here 😅 )

What do you mean by full fetch?

We add an Etag approach,

Ok, will we use ETag and If-None-Match headers to detect for changes?

We consider if we can make it more optimised (by stripping data?) out of payload doing incremental updates as discussed in the past.

Not sure if I understood entirely, what do you mean by incremental updates?

This issue is about doing this in 2 completely different places, with two completely different Frontend implementations.

List view is built with Vue, show view is built in haml.

@filipa we should absolutely avoid making http requests if the user is not actively on the page. Thankfully we have a nice function for that 😃

Incremental updates is via a diff. Here we would have to do some deep merging of data. Not a big deal. I have the code on a branch somewhere. It never got merged because there we a few bugs and the eTag solution is more elegant from a Database hit perspective.

I am working on a realtime module for VueResource. Once we get feedback from the team about the current etag solution being used, I can move forward with Realtime Title updates and that will get merged.

Also for pipelines I still think every 10 seconds is better than 3 seconds. 🎉

Why not push pipeline updates from backend to client via WebSockets?

@gaui Scaling 😃

Evaluate different solutions for real-time updates

And this is what we ended up considering: eTag

@gaui Websockets and event-based will happen in next couple of months, probably Q3 or Q4.

@ayufan perhaps this issue can be improved title wise.. or is this the meta issue.. that will take on both real time pipeline lists, as well as real time pipeline graph

@selfup will the pipeline list feature real time mini pipeline graphs?

assigned to @filipa

@dimitrieh the one in the table already is, the one in the MR won't be addressed in this issue. :)

@ayufan aren't we using etags for realtime? Does it matter if backoff or just regular polling is used?

For %9.1 we will focus on shipping real-time updates for Pipelines List that is present in:

Pipelines,
Merge Request Pipelines,
Commit Pipelines.

For %9.2 we will try to ship Pipeline Graph update in real-time, rewritten in VueJS. Still to consider if we should do the same for table view.

Discussed with @ayufan and @jschatz1:

For 9.1. Let's focus on Pipelines List realtime update by using backoff pooling.

@jschatz1 As discussed:

We are doing backoff with short backoff limits

We will add active tab logic code before 9.1 for the backoff code.

mentioned in merge request !8357 (closed)

@filipa @smcgivern @selfup

After talking with @andrewn here is the plan.

Don't do backoff. It doesn't work.
They (Gitter) have written the code for active tab switching. We can use theirs.

@pcarranza @bkc @jschatz1

we already know how we are going to handle doing this

but for the #infrastructure team we need to consider the following:

Feature Flag Things

we have to set a feature flag.

if the feature is deemed too heavy, the endpoints will return a 403 instead of a 304 on subsequent calls.

this will be caught by the catch clause in the poller and stop polling

initial api call will return 200

then we can turn it back on (304)

so we can just throw errors (instead of big red flash messages) until we are comfortable

then once we are comfortable we switch to flash messages and not return 403

@jschatz1 @filipa here is the working ShortPoller:

https://gitlab.com/gitlab-org/gitlab-ce/blob/f819fac73556d703952b47247d528a693c3c0130/app/assets/javascripts/vue_poller/index.js

@selfup the concept lgtm

a couple of notes:

use harmony syntax
use a constructor without side effects
move Vue.use(VueResource) outside the class
we don't need to instantiate Date to get current timestamp, we can use Date.now() instead
We could make pooling into isPooling to make it more obvious of what it means does
In the setInterval, I don't think we need to return null from looking at the documentation, we could simplify it to

this.state.polling = setInterval(() => {
  if (this.state.isPolling) {
    this.fetchData()
  }
}, this.options.time)

This creates one less branch

since this is a class we don't nee to provide this.options as a parameter, we can access it in fetchData, and instead of

 fetchData({ url, data = {}, success, error }) {

we can do

fetchData() {
 const { url, data, success, error } = this.options.
}

When merging two objects we should merge into a new one instead of changing the first one. In here:

Object.assign(data, {
      headers: { 'X-Last-Fetched-At': lastFetchedAt },
    });

we are mutating this.options.data and we should avoid mutating data.

If we open a MR we can move this comments there. You can see documentation for my notes in this issue and in this MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9961 https://gitlab.com/gitlab-org/gitlab-ce/issues/28713

mentioned in issue #28984 (moved)

@ayufan is backend ready for this?

added ~901060 label

@filipa Pipeline details page would probably require some additional backend work.

mentioned in merge request !10210 (merged)

mentioned in merge request !10223 (merged)

assigned to @godfat

@grzesiek it requires a lot Frontend work. 😂 we are only targeting index page now :)

@grzesiek @godfat for %9.1 this is only about pipelines index page. I will open a new one for show page and update the description.

changed title from Update pipelines in real-time or at least refresh to Update pipelines table in real-time

mentioned in issue #25226 (closed)

Updated the description. Issue about pipeline show view is: https://gitlab.com/gitlab-org/gitlab-ce/issues/25226

Makes sense, thanks @filipa 👍

ok, I think I have caught up reading the discussion in #26926 (closed) and the implementation in !9036 (merged). Special thanks to @andrewn and @adamniedzielski, very clear explanation. I'll start working on this tomorrow.

mentioned in issue gitlab-com/organization#69

closed via merge request !10210 (merged)

mentioned in commit maxraab/gitlab-ce@5b2b42a4

reopened

removed assignee

changed milestone to %9.2

removed Deliverable label

Should we in %9.2 continue our work with making this time Pipeline Graph to be real-time? These changes would help us when visualizing Cross-project pipelines, as we would prepare everything to show in much more efficient way.

@ayufan Yes! :)

This should be closed shouldn't it? 🤔

The rest of store is covered by: https://gitlab.com/gitlab-org/gitlab-ce/issues/25226. Closing this one.

closed

changed milestone to %9.1

added Deliverable label

Update pipelines table in real-time

Single Source of truth:

Description

Proposal

Links

Designs

Child items 0

Activity

Feature Flag Things

Admin message

Admin message

Update pipelines table in real-time

Single Source of truth:

Description

Proposal

Links

Activity

Feature Flag Things