Geo DR: dashboard improvement brainstorm

Description

Here are the current screenshots of the dashboard to manage Geo and DR:

What could we do to make them better? Let's use this issue to brainstorm. Once we'll get a better idea of what we want to accomplish, we'll create separate issues for them.

Proposal

1. Indicate the reason why a repository failed to sync
2. We only mention repositories synced status. Can we add % files replicated as well?
3. Should we allow to remove the primary node?
4. How does it look when we detect a health problem?
5. Is there a button to add a new node? How do nodes get detected?
6. Show date of last time synced (I have the feeling it would be reassuring)?
7. Show if a sync is currently in progress

@rspeicher @dbalexandre @stanhu I'd like your thoughts on those points.

Yes, we should add the % files replicated, perhaps break it down into specific types:

• LFS objects
• Attachments
• CI artifacts etc.

If we could also list which files or repos are being copied at the moment, that might be helpful.

Other things to include:

1. Current Geo Node (if you're looking at the secondary, it should be clear that GitLab identifies which secondary you are, e.g. row 2 in that list)
2. Diagnostics for each node: database replication status, SSH clone, Geo API success/failure, error messages, error codes

Indicate the reason why a repository failed to sync

Definitely.

We only mention repositories synced status. Can we add % files replicated as well?

I'm curious how the Primary knows the status of the secondary's sync, since it's the secondaries that have the tracking database. Do we have (or have plans to add) a method for the secondaries to send back a kind of progress ping?

Should we allow to remove the primary node?

To what end? Wouldn't that disable Geo entirely? Or would it randomly pick a secondary to become the new primary? How does that new primary notify the other secondaries?

How does it look when we detect a health problem?

One of those screenshots seems to indicate it: "Could not connect to Geo database". That said, that doesn't stand out at all, and doesn't even indicate that it's a problem. As we get further along with a working DR solution we'll likely need some UX and Frontend love here.

Is there a button to add a new node? How do nodes get detected?

Right now since so much of the process is manual I don't think it's feasible. This will depend on Stan's suggestions to automate a lot of this process first.

Show date of last time synced (I have the feeling it would be reassuring)?

Show if a sync is currently in progress

@rspeicher

To what end? Wouldn't that disable Geo entirely? Or would it randomly pick a secondary to become the new primary? How does that new primary notify the other secondaries?

Well, there is currently a button that lets us remove the primary node (thanks to this screenshot). I assume we change remove it from the UI, @dbalexandre

To what end? Wouldn't that disable Geo entirely? Or would it randomly pick a secondary to become the new primary? How does that new primary notify the other secondaries?

Well, there is currently a button that lets us remove the primary node (thanks to this screenshot). I assume we change remove it from the UI, @dbalexandre

@regisF If we remove this button from the UI we should add a button to allow an admin to edit the primary node in case that he entered some wrong information or it has been changed. Wdyt?

@dbalexandre alright, so the main use case intended for this button, is to actually designate a primary node?

If that's the case, we should instead have a button saying Make primary or something.

By the way, @rspeicher @stanhu what did you have in mind in case of actual disaster, to designate a secondary node as the primary node? We can't the UI in this case.

@regisF That's a good question. We don't have a good system right now. In that case, what needs to happen?

• Secondary needs to change PostgreSQL DB settings from primary (disable read replication)
• In admin panel, Geo needs to be disabled on secondary (or secondary needs to be promoted to primary, and primary demoted)

@brodock Anything else?

@stanhu shouldn't it be done at the CLI level? I mean, there won't be any GUI available, I guess.

No matter what, we need to keep track of the status of the replication (in %) at the secondary node level. Why?

Imagine the scenario: we have 2 secondary nodes (S1 and S2), and one primary (P). If S1 is at 50%, and S2 at 70% of the primary, and if primary crashes, users will need to designate a secondary node, and they should be able to pinpoint which node has the most up to date information and % replicated data.

I think we are kinda doing that at the moment, but are we really?

@regisF how to manually promote a secondary node into primary can be found here.

This is how it could look like. The wireframe shows 3 use cases:

• sync in progress
• sync completed with no errors
• sync with errors

That reminds me that we should notify administrators when sync fails (https://gitlab.com/gitlab-org/gitlab-ee/issues/1816).

@stanhu what's the most important thing we should have from the mockup above?

I think we should focus on getting the different numbers first. Like displaying the % of sync for each type of thing we replicate.

Then we can focus on other details. If you agree, I'll open different issues with a concrete proposal.

Closing this issue as we will move forward with another design anyway. Thanks.

closed

mentioned in commit c65634cf

mentioned in commit 33699166