Geo: Consider supporting writes to primary
In !1283 (merged), we are adding support for multiple read slaves that can write to a single master. The commits will be synchronous so that a change in the master will only take effect if all secondaries have committed the change.
What if we made GitLab Geo use this functionality across different datacenters? Here's how it would work:
- Let's say we have a primary Geo node in US East and secondary in Europe West.
- A user accessing the site in Europe West would be talking to an Geo node in Europe, with a local read-only DB there as well.
- The application would send any writes to the US East database.
Now this may not work well in practice if Rails sends a DB update for every page visit, but in theory reads should dominate the load.
This will likely slow down write performance since multiple nodes have to be involved, but it should simplify the read-only issues we have to deal with in gitlab-org/gitlab-ee#1744 while also making the Geo node a fully-functional version of GitLab. We would also have to find ways to propagate attachments and other files faster.
Thoughts?
Activity
mentioned in issue #1744 (closed)
I believe Synchronous replication will not work for Geo. For some context on "real use scenarios": https://gitlab.slack.com/archives/geo/p1487695629000019
Few things come to my mind:
- If we make all nodes use synchronous replication, whenever connection between any of them drops, we break the entire "cluster" and no more writes can be done.
- Whenever one node is in a poor connection or high latency this will impact and slowdown for everybody
@brodock Good points. Even with customers who want active-active support (at least for git repos), the commits would have to reach a majority of nodes before they are accepted, so the poor connection/high latency case is less of a concern than a datacenter disappearing overnight.
assigned to @regisF
assigned to @rdavila
