GitLab Geo (Read-Only secondary servers) (EE Option)

More and more customers are asking for this, as Job already mentions. Just this week I've heard from 2 so far, and have a conversation scheduled for tomorrow about the same.

The routing piece @sytses mentions is very intriguing and would cover many use-cases. If users can simply point at their 'local' instance and have it seamlessly forward requests to the active cluster it would be great.

A challenge I see is that customers already think we have a magic HA configuration for local HA clusters. They don't want to configure an external HA database, HA Redis, or an HA filesystem. In many cases they don't know how, nor do they have the resources to maintain it. I see this problem only being increased with global HA. How do they configure the database and filesystem replication? These don't sound trivial.

@dblessing HA in the same datacenter is a different problem. But good point, I made https://gitlab.com/gitlab-org/gitlab-ee/issues/77 for that.

Title changed from GitLab RE (Slave servers) to GitLab RE (readonly secundary servers)

Other solutions for cloning faster locally:

Git fastclone https://news.ycombinator.com/item?id=10580612
Git Webhook Proxy https://github.com/orrc/git-webhook-proxy
GitLab mirrors https://github.com/samrocketman/gitlab-mirrors

mentioned in commit faa08fe6

Title changed from GitLab RE (readonly secundary servers) to GitLab RO (readonly secundary servers)

I changed the name to GitLab RO instead of RE because:

It is easier to remember since these are ReadOnly servers
It has two letters of difference to CE instead of one

@sytses haha, great. RE was confusing.

Title changed from GitLab RO (readonly secundary servers) to GitLab RO (Read-Only secondary servers)

Added ~139512 label

Added ~139607 label

So in theory setup of RO server should be as easy as:

checkbox to turn app in read-only mode
provide URL to main server
probably add RO public key to main server for git sync
setup slave postgres on RO

It would be nice that if you push to GitLab RE it forwards your request to the master server.

Agree. But might be hard to do from start. I propose to leave it for next iteration

According to application changes we can start with next:

UI in application settings that allow you to configure app either as master or as RO
Make worker for master that push to RO server(s)
Make some middleware that reject any POST/PUT/DELETE requests in Rails app if RO enabled

When I think about attachments - for now we can just point it to master server so we dont need to sync it. It might require just single change in Markdown library

@dzaporozhets Thanks for your comments.

I agree that this setup of the RO server should be as easy as you describe.
I'm OK with leaving the forwarding of pushes to master for the future.
I assume that if you change the application settings you have to restart, or not?
Do we need a worker for master or can we just use a system hook?
The attachments can point to master indeed if they are images, non-images require authentication I think.

I assume that if you change the application settings you have to restart, or not?

@sytses depends on implementation

Do we need a worker for master or can we just use a system hook?

we need a worker. System hooks has nothing to do with pushing. Even if you trigger system hook on each push we should anyway implement logic that does push from master or fetch from slave. I think its easier to just make a separate worker for master. We might want to re-use this worker in future for 2-side mirror feature.

The attachments can point to master indeed if they are images, non-images require authentication I think.

I think we don't require authentication for attachments.

I don't understand what this worker looks like. Does it periodically push every repo? What kind of period are we talking about?
http://doc.gitlab.com/ce/security/user_file_uploads.html says "Note that non-image attachments do require authentication to be viewed." (which I find strange, but if we don't require authentication for attachments we should update that)

@dzaporozhets ^^^^

@dzaporozhets answered 1. during a talk, the primary will push every repo when it is updated to all the secondaries. This is simpler because it has one action, no notify and pull. We can't use existing system hooks since they don't fire on push. Dmitriy also said it enabled two way mirror (can't that also happen with notify pull?) and the secondary triggers and activity stream would be updated normally (I probably misunderstood since the activity stream is in the readonly database. Regarding 2. he thinks both attachments work the same (no-auth) so we should update the documentation.

I asked @dblessing about the non-image attachments in https://gitlab.com/gitlab-org/gitlab-ce/commit/63a1a581e937ff6d21e7e6ca4774b7907c6a0c1b#note_3084491

Images are still the only attachment type that doesn't require auth. @DouweM tested this behavior when he helped me with the security doc.

@dblessing thanks

@JobV proposed to name it GitLab Geo, I think that is a better name, @JobV will execute the name change.

Title changed from GitLab RO (Read-Only secondary servers) to GitLab Geo (Read-Only secondary servers)

during a talk, the primary will push every repo when it is updated to all the secondaries. This is simpler because it has one action, no notify and pull. We can't use existing system hooks since they don't fire on push. Dmitriy also said it enabled two way mirror (can't that also happen with notify pull?

@sytses it can but only for case when both instances are GitLab. If we want GitLab to mirror on kernel.org or github or bitbucket we want to make push. Anyway we will discuss it one more time with developer on stage of implementation. The simplest solution will win.

and the secondary triggers and activity stream would be updated normally (I probably misunderstood since the activity stream is in the readonly database.

my bad. you are right. With read-only database it wont be a case.

@brodock will work on it with my guidance. First step to do:

Get 2 gitlab vm running (one on db master and another on db slave)
Implement a bit of logic for GitLab when each instance can say: “I am slave, my master is master.gitlab.com" or “I am master, my slaves are ..."

cc @sytses

Only after first step is working we can start thinking about repository sync

Reassigned to @brodock

mentioned in merge request gitlab-development-kit!96 (merged)

@dzaporozhets I like that having the master push the repo will enable mirroring to non-GitLab instances, the issue for this is in https://gitlab.com/gitlab-org/gitlab-ee/issues/116

@JobV this has milestone 8.5 here but 8.4 on https://about.gitlab.com/direction/ I really hoped for this to ship in 8.4 but probably missed setting the milestone

@JobV I called with Dmitriy. Both primary and secondary will be run with the EE codebase, so it is very hard to ship this in 8.4. We'll move it to 8.5 https://gitlab.com/gitlab-com/www-gitlab-com/commit/775085df281d6cae47cc63598af4e1963ba48814

@brodock please create WIP merge request as soon as you have something to push. Put a link to mr here

mentioned in merge request !112 (merged)

@sytses @dzaporozhets shame. Thanks for updating and organising. I'll make sure to double check whether /direction items are properly set to the corresponding milestone.

mentioned in commit gitlab-development-kit@85909838

mentioned in merge request !139 (merged)

mentioned in commit 27e2d103

Update https://gitlab.zendesk.com/agent/tickets/14580 once we know exactly which release this will land in.

mentioned in commit 7efdde6b

What we've done

PostgreSQL replication worked well
Redis is sharing session (primary <-> secondary)
Authentication should happen on the primary server because it changes a lot of states (most related with security measures, like brute force prevention, password recovery, etc).
If this becomes an issue, there is an alternative to store the states in Redis, but will take some time to implement.
We have a middleware to prevent potentially writing operations on secondary servers (to make sure its readonly)

Next steps

How we are going to update repositories on secondary servers:

I had a call with @dzaporozhets today, and we discussed about the following approaches:

Primary should push to all secondaries
Each Secondary should pull from primary

The conclusion is that if Primary can push to any secondary, then can any other user. To overcome this, we will have to break our permission system.

It's easier to have Secondary servers pulling from Primary, and it also makes easier to deal with connection issues, as primary doesn't have to keep track of that.

We are already replicating PostgreSQL from Primary to Secondary, and we do need to replicate Redis to share session data, so it's the perfect place to store data and coordinate secondary repositories updates.

We don't want to do "cron like" pooling of every single repository, but update only the modified ones.

The idea we discussed is to create one queue for each instance in redis, where primary will enqueue repositories that secondaries needs to fetch.

This queue should be namespaced with something like geo/[:secondary_host]. We can use redis MULTI to bulk update every namespace in a single transaction, to make sure we don't have inconsistencies between secondary instances.

As we don't want to require Procfile updates to run Geo, we will try to use Sidekiq-cron to pool from the repository updates queue. As a starting point we will pool every 10 seconds, but will try to make it down to 5.

mentioned in issue #276

@brodock please write down progress since last comment https://gitlab.com/gitlab-org/gitlab-ee/issues/76#note_3515028

mentioned in merge request !179 (merged)

@dzaporozhets: I'm sorry I was a little off the radar this week. I had some unplanned personal life issues to solve, I will catch-up on the weekend and be on schedule again.

The original idea to use Redis as a communication bus from primary to secondary will not work as expected for the queues, as we can't write on secondaries (I initially had the impression we could have master-master replication), and the latency with them could make things complicated to have secondaries connecting to master redis directly.

So there is the need for a fix on the idea.

What I come up with is, we can still use redis to buffer updates to a queue, and then submit a single HTTP request from master to each secondary, with all project ids that have their repositories updated recently (we consume the queue to get thous ids).

On the secondary nodes side, when receiving this request, it will enqueue sidekiq jobs to fetch repositories/changes.

MR to track changes: !179 (merged)

Will this ship with 8.5 or 8.6?

@dblessing Working beta in 8.5, more reliable version in 8.6

@brodock ok thanks for update

@brodock after repository sync we need to

documentation
handle attachments (patch markdown to use master host for attachments url in markdown)
make a demo to service engineers with staging server?

cc @sytses @DouweM

make a demo to service engineers with staging server?

Milestone changed to 8.6

@brodock Can you give a status update? I notice https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/179 isn't merged yet and this didn't end up going into 8.5.

@DouweM I made the documentation on friday (forgot to push, will do it now). After I setup the demo instances, it became pretty obvious that the current (hacky way) authentication wasn’t working as expected (we were sharing session using redis).

During development, as it was on the same machine everything worked fine, but sharing a remote redis is far from good and requires actually two redis instances on the secondary nodes to work (one for the session and another for sidekiq).

So I’m moving forward to the OAuth authentication, which I was going to do anyway as a next step.

There will be an “OAuth application” registered for every node, and secondaries will authenticate based on that. I'm not using omniauth but plain OAuth client in a similar way old CI worked. I decided to go this route without omniauth because the integration with devise does not provide an obvious way to dynamically define a provider.

I will deploy OAuth in a separated branch based on the !179 (merged) to have the demo instance working and make the final tests. We can release after that.

@brodock Thanks for the update! Is it an option to use (something similar to) gitlab-shell's .gitlab_shell_secret (http://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example#L413), and skip OAuth? Every machine will already have that. It's currently used to authenticate between gitlab-shell, which terminates SSH connections, and the internal API, which verifies user access to repositories.

Milestone changed to 8.5

@DouweM Not really, for the user authentication, the problem is reliably sharing the state "user X is authenticated" between instances in the context of a user session.

OAuth is perfect for that. If I use otherwhise my "own protocol" I will end up implementing OAuth from scratch which is not a good idea.

@brodock Ah, right, we want to authenticate the individual user. OAuth makes sense.

You can keep using the secondary server if master is down if you're already authenticated.

mentioned in merge request gitlab-shell!41 (merged)

mentioned in commit gitlab-shell@0261a26e

mentioned in merge request !236 (merged)

mentioned in merge request !237 (merged)

mentioned in issue #356 (closed)

mentioned in issue #366 (closed)

mentioned in issue #367 (closed)

mentioned in commit d71a68ee

mentioned in commit df8996c8

mentioned in issue #371 (closed)

mentioned in merge request !251 (merged)

mentioned in merge request !252 (merged)

mentioned in commit 27c85c76

mentioned in commit 1e79d74b

mentioned in commit b3e45526

mentioned in commit afaf6b78

mentioned in commit 197b1954

mentioned in merge request !267 (merged)

mentioned in merge request !274 (merged)

mentioned in commit 9006f046

mentioned in commit c0337aa7

@brodock Do we now have issues for everything Geo related that still needs to happen? I see https://gitlab.com/gitlab-org/gitlab-ee/issues/371 for syncing SSH keys, but I think there is some other on-disk data that potentially need to be synced, like uploaded files and avatars, build traces, build artifacts, LFS objects. I think we should have an issue for everything Geo related that still needs to happen; If it's not in the issue tracker, but just in people's heads, it doesn't exist. :)

Can you also link to all Geo related issues from a task list in the description of this issue, so we can see at a glance what parts of Geo are done, and which aren't yet?

mentioned in commit 07fdc10d

mentioned in commit 20a3715e

mentioned in commit b77103a7

mentioned in commit 9284c32a

mentioned in commit 645e5d16

mentioned in merge request !282 (merged)

mentioned in merge request !287 (merged)

mentioned in commit 891b27c0

mentioned in commit 9c60074d

mentioned in commit ba623419

mentioned in commit 4e883231

Marked the task Geo: SSH keys Sync (https://gitlab.com/gitlab-org/gitlab-ee/issues/371) as completed

Milestone changed to 8.7

mentioned in issue #414 (closed)

mentioned in issue #415

Git LFS support / synchronization (gitlab-org/gitlab-ee#415) is blocked by gitlab-org/gitlab-ce#13825

We should discuss what / how to do with the following items:

Geo: Git LFS support / sync (gitlab-org/gitlab-ee#415)
Geo: Build traces Sync
Geo: Build artifacts Sync
Geo: Uploaded files and Avatar Sync

I believe syncing local files are not something we should do using REST requests and sidekiq (which is what we used for repository synchronizations).

There are two different approaches that could help us here. In gitlab-org/gitlab-ce#13825, we will use S3 storage object compatible daemon to handle LFS support, and we are also experimenting with CEPH, which also supports geographical replication (http://docs.ceph.com/docs/master/radosgw/federated-config/).

Getting everything under either CEPH or S3 compatible object storage, will solve them. cc @dzaporozhets @DouweM @JobV @jacobvosmaer

mentioned in merge request !302 (merged)

mentioned in commit 6c7221b6

mentioned in commit 357e84a7

Marked the task Geo: Display Attachments from Primary node (gitlab-org/gitlab-ee#414) as completed

@brodock I don't know much about CEPH so I probably leave this up to @jacobvosmaer for advice.

@brodock read-only LFS support should not be too difficult to make possible. To replicate the files I would use periodic rsync jobs, I think we can get away with that.

If we cannot get away with rsync then sticking all of those resources in blob storage (like S3) is also feasible, but it would be more work.

@jacobvosmaer rsync might be good only for small repositories with infrequently updates as it can take ages to even start the process if you have a bigger one.

what we current replicate are ~10 seconds only behind primary, if we go the rsync route it would be minutes or even hours, right?

@brodock I thought we were talking about the uploads/artifacts/builds/LFS blobs, not repositories.

I'd be happy to brainstorm with you but a call might be more appropriate than these comments.

@jacobvosmaer when I mentioned repositories I was thinking about LFS, sorry for the confusion. In my university we used to store linux .iso images and big files in git repositories (which was a bad idea but a perfect use-case for LFS).

@brodock OK :) I agree there will come a point when just starting up rsync for the (global!) lfs-objects directory of a GitLab server becomes painful. I think the appropriate solution for that, when that time comes, is to store LFS objects on S3 (or something like it).

I asked @dzaporozhets to look into what we have now. I'm concerned about the following:

We have documentation https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/236/diffs but it doesn't detail how Geo works internally, I think due to the complexity of the feature this is essential.
I do not see a testing plan. I propose that Gabriel works with Pablo to combine this testing with Ceph testing, as proposed in https://gitlab.com/gitlab-com/operations/issues/1/#note_4527326 (please note that this isn't about LFS, Ceph is just used as storage for the repo's themselves)
At the start of this issue I proposed using a system hook https://gitlab.com/gitlab-org/gitlab-ee/issues/76#note_3049586 This would be a new system hook for push. At the time we didn't do this since we were thinking about pushing the data completely, but later on we switched back to 'send notification to secondary and then have secondary pull'. Is it possible to use a new system hook after all?
In the current limitations https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/236/diffs#23d8dcf76919dfcf001c0f0b3946701640c27f02_0_66 we don't list that the primary has to be online in order for people to log in via OAuth (but that existing sessions are not affected by primary going down)
There is a batched operation (every 10 seconds https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/236/diffs#23d8dcf76919dfcf001c0f0b3946701640c27f02_0_101 ) but I don't understand why this is needed. Any delays might be unacceptable to customers.
All notices of master/slave (such as in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/236/diffs#23d8dcf76919dfcf001c0f0b3946701640c27f02_0_56) should be replaced with primary/secundary
Instead of having MySQL as a TODO in http://doc.gitlab.com/ee/gitlab-geo/database.html can we just say that we require PostgreSQL for this?

@sytses

Yes we should add documentation for devs & service engineers too
@brodock please create operations issue for testing.
Maybe. I need to talk to @brodock first
@brodock please update documentation
Maybe. I need to talk to @brodock first
@brodock please update docs
+1 for making it only for PG

I asked @DouweM and @brodock to schedule a call together to discuss current architecture and what is next step here.

I will update this thread with additional information after call

Summary of call with @DouweM and @brodock:

I will lead development of the Geo feature
@brodock will write code
@DouweM will do code review

Summary of call with @brodock:

Documentation improvements need to be done based on https://gitlab.com/gitlab-org/gitlab-ee/issues/76#note_4746360
We make Geo feature only for PostgreSQL
Avatars, LFS and builds artifacts should be synced with one technology. We can start with rsync and then consider some replicated FS
SSH keys sync logic for primary server should use System hooks
We consider use system hooks for repository sync. This means we need
- implement push event for system hooks (CE)
- use system hooks for Geo feature in EE

Advantages of using system hooks to inform secondary server about new ssh keys or git push:

minimal amount code needed (creating system hook per node in database)
single logic for repos and keys
we re-use existing feature (system hooks) that is well tested and used by people in CE and EE

Disadvantages:

sidekiq job on every push multiplied by amount of secondary servers. If instance has 10 pushes per second and 2 secondary nodes it will be 20 sidekiq jobs per second.

cc @sytses

Push data from GitLab.com now (Monday 7:30 UTC).

irb(main):001:0> Event.code_push.where('created_at > ?', Time.now - 12.hours).count
=> 41530
irb(main):002:0> Event.code_push.where('created_at > ?', Time.now - 1.hour).count
=> 2957
irb(main):003:0> Event.code_push.where('created_at > ?', Time.now - 5.minutes).count
=> 241

Basically around 1 push per second. That mean if GitLab.com had Geo enabled with 2 secondary servers it will be 2 sidekiq jobs per second.

cc @brodock

@brodock I just had a call with @sytses . We will go with simplest possible way (does not mean its a right way to do things). That means:

we will use system hooks for everything (simple, minimum code)
we don't do sync for avatars/artifacts for now.
as soon as we have Geo based on system hooks we just deploy it to testing env (you mentioned you already have one)

I will setup a call with you tomorrow to discuss all this

From sidekiq readme https://github.com/mperham/sidekiq#sidekiq throughput for v4 is around is 4500 jobs/sec

So if bandwidth and hardware is good we can expect idea with system hooks works to certain scale.

Plan for next 2 weeks:

Tuesday - Use System hooks for ssh keys sync
Wednesday - Review System hooks for ssh keys, merge and deploy to Geo test server
Thursday - Push events for System hooks in Community Edition
Friday - Review Push events for System hooks in Community Edition, merge it and merge CE into EE. In meantime do documentation updates
Monday - Repository sync via System hooks
Tuesday - Cleanup and write engineer doc about architecture
Wednesday - Deploy to test server and test it
Thursday - Present it to other people

cc @sytses @brodock

@dzaporozhets thanks, sounds great!

OK so we need proper way to check to authenticate requests from system hooks to our API. We will implement part of https://gitlab.com/gitlab-org/gitlab-ce/issues/13478 and then backport it to CE.

@brodock here is a plan:

add token field to hooks table
fill this field with hash when generate system hook for geo node
add this field to request header of webhook (only if present) so in API enpoint we can confirm that request comes from legit primary server.

After its done we update !332 (merged) with new auth and merge it.

mentioned in merge request !332 (merged)

mentioned in merge request !334 (merged)

I've been thinking about this issue a lot lately, particularly with my main project I am working on.

It would be really good to be able to synchronise the whole DB between a local instance and a remote 'master' instance.

For me the solution would seem to be the following.

Each local instance of a gitlab install has a 'special' tag or prefix for all the issues, comments, files etc that are submitted.

for example: I have my personal laptop, I would commit all my stuff locally and tell it to prefix (or postfix) all my stuff with my username, and PL (to signify my personal laptop) Then at my office, I can have the same setup, but with B (for bureau)

Now when I make a comment, the system will be able to distinguish all my comments etc, and there should be no clashes with others.

by tagging / linking my localy made comments into the principle (parent) issue / comment, it would seem reasonably that any conflicts would be minimal. If there are any conflicts due to the nature of the postfix detail the server will be able to determine who is responsible for 'merging' stuff into the main master repository.

I don't feel that this should be a huge required modification in the system, the prefix could even look like a tag ~ or maybe it could be a tag, and could be added automatically.

After all, all the comments, issues, commits are already 'tagged' with the user who up sent them, so having the PK in the db to cross all these fields shouldn't be an issue.

I have a feeling that just the username would probably work, provided people always work on their local system only, and it regularly syncs to the main parent.

Appologies for the long post.

Davem

mentioned in commit 940bcc7b

Test server:

http://gabriel-ee.gitlap.com / http://gabriel-geo.gitlap.com (geo is in AMS). Credentials are in 1password (support vault)

Progress so far:

we use system hooks for ssh keys
we implemented optional token for system hooks to verify request (will be ported in CE)
we have test server running

Next step:

Trigger System hooks for post-receive events in CE (we need it for repo sync)

cc @brodock @sytses @DouweM

mentioned in commit 2981044f

We will be 1 day behind original schedule because correct implementation of push system hooks need some extra work.

@dzaporozhets @brodock Great work

I had a call with @brodock. He will finish https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3744 with my recommendation and then continue with Repository sync via System hooks.

mentioned in merge request !347 (merged)

Once https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/347 is merged we have both repository and ssh keys synced via system hooks. Next steps are:

Today: deploy to test server + update documentation
Wednesday: QA on test server and share knowledge
Thursday: Fix bugs if any to be sure its ready for Friday release

After 8.7 release:

Backport safe webhooks to CE
Investigate possibility to sync wiki via system hooks

cc @brodock @sytses @DouweM

mentioned in commit a9e4af35

mentioned in merge request !350 (merged)

mentioned in commit 6f9a97aa

mentioned in commit b6608854

I will integrate the license add-on after documentation (today)

Geo staging is deployed and ready for testing. Contact me or Gabriel for testing URL

I'm seeing this when I access the secondary:

It makes sense since we use OAuth, but can we make that transparent, or make it look nicer?

Seeing error 500 on http://gabriel-geo.xxxxxx.com/gitlab-org/gitlab_git

When I try to sign out on the secondary, I see:

It makes sense, but it would be great if signout worked :) Either by redirecting to primary to sign out, by directly deleting the session cookie for the secondary, or by showing a link to the primary.

Also, "Gitlab Geo" should have "GitLab"

@brodock To fix:

500 error on http://gabriel-geo.xxxxxx.com/gitlab-org/gitlab_git
Cant logout on secondary server (even if logged out on primary)

Found what is causing the 500 error on geo: Gitlab::Application.secrets.db_key_base is different on both machines. For some reason a few repositories work just fine while others tries to #external_import? and triggers project.import_data.credentials which crashes with:

OpenSSL::Cipher::CipherError: bad decrypt
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:73:in `final'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:73:in `crypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/encryptor-1.3.0/lib/encryptor.rb:44:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:197:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:280:in `decrypt'
        from /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/attr_encrypted-1.3.4/lib/attr_encrypted.rb:143:in `block (2 levels) in attr_encrypted'

This is something we have to document.

@DouweM how are you logging out? I can logout just fine with my account.

Very nice! Replication seems very fast.

I tried to edit settings to see what it would tell me. In my case it spun forever in 'Saving'. Only when I hit another button finally did it redirect again and show the 'You cannot do writing operations on a secondary Gitlab Geo instance' error.

Can we show this banner preemptively when a user goes to any edit page? Since there's otherwise no indication you're on a secondary node it can be confusing until you see that error.

@dblessing I have a different idea to propose... I think we could add a "geo-secondary" class to body and disable forms (with few exceptions). What you think? (idea for after 8.7)

I pressed the OAuth buttons but I was unable to login.

@sytses does it gives you any specific error message? You can use the password in the Support vault to get access if OAuth isn't working for you.

@brodock Sounds like it would be a good solution

mentioned in merge request !354 (merged)

mentioned in commit 09100b61

@brodock I'm signed in on both primary and secondary, and hitting the sign out button in the top right on the secondary. Nothing special going on :)

I think we could add a "geo-secondary" class to body and disable forms (with few exceptions). What you think? (idea for after 8.7)

I dislike doing this just using JS. Although adding if Gitlab::Geo.secondary? blocks all over the place is not a good idea either...

Changes made by !354 (merged):

Replicates repository removal (using system hooks)
Replicates repository rename (using system hooks)
Replicates repository transfer (using system hooks)
Whitelisted Sidekiq routes so you can use web interface in a secondary node.
Change "Gitlab -> GitLab" for the readonly error flash
Check license add-on to allow any Geo features.
Do not execute on a secondary node: StuckCiBuildsWorker
Do not execute on a secondary node: HistoricalDataWorker
Documentation changes: (master, slave -> primary, secondary)
Document we require db_key_base(secrets.yml) to be the same in all nodes

mentioned in commit 97e7080d

Next step for 8.7:

wait for !354 (merged) to be included in patch release.

For 8.8:

Make another user testing on staging server with latest fixes
Fix most of the outcome from such testing
Backport to CE and finish safe web hooks feature (add UI part)
Decide on wiki repositories sync

cc @brodock @DouweM

mentioned in commit a193fde8

I had a call with @brodock. Schedule for this week:

Tuesday - deploy latest Geo to staging and make user testing with one of the team members
Wednesday - more testing + fix possible issues
Thursday - safe web hooks feature to CE
Friday - another call to discuss wiki sync implementation

mentioned in merge request !364 (merged)

mentioned in commit 8281793e

mentioned in issue #522 (closed)

Milestone changed to 8.8

mentioned in merge request !366 (merged)

@brodock current status please

@dzaporozhets

Found a problem acessing merge requests with "unchecked" merge_status in a secondary geo (see: !366 (merged)) and logout isn't triggering flash errors for @DouweM anymore, but we are not logging out the user from their primary node (we don't do single sign off yet, ticket to implement it with a simple proposal: https://gitlab.com/gitlab-org/gitlab-ee/issues/522)

Webhooks backport will be ready today: gitlab-org/gitlab-ce!3940

@brodock please follow next plan:

finish webhook backport and get it merged into CE
List of current knows bugs with Geo feature and estimate for fixing those. (should be a list in the issue description with links to merge requests)
Fix current knows bugs
Make another user testing with one developer and one service engineers
Make a list of found bugs based on recent testing and fix those
Prepare a load testing for staging server to find out point of failure
Inform @dzaporozhets and @DouweM that feature is ready for use (in alpha state) with a customer

cc @sytses @stanhu

mentioned in commit 10743b27

mentioned in commit 30bca296

mentioned in commit 86cf583a

mentioned in merge request !380 (merged)

@brodock @dzaporozhets can you give an estimation for this? It doesn't matter how long it takes, but we (I) need to make sure this gets delivered to customers, the sales team and marketing properly.

TODO for @brodock based on call:

Will provide estimate on alpha version today
Finish single sign out and deploy to staging
Ask few people for user testing (I proposed @JobV and @vsizov)
Makes sure license with Geo add-on can be generated (I will double check it later too)
Makes stress testing
Cooperate with @JobV for delivering/marketing

we (I) need to make sure this gets delivered to customers, the sales team and marketing properly.

@JobV

@JobV our target is to get this in GA for 8.8... we had no major issue during last test phase, so I'm confident that whatever we may find on the next ones are more in cosmetics/usability area, that we can quickly fix and don't delay anymore.

@dzaporozhets @brodock

@brodock status update please

Created / added some missing issues to the summary.

Single Sign On still has some things to fix on the code, will handle that tomorrow. There is a proposal for the benchmarking (stress testing): #560 (closed). I will ambush @JobV and @vsizov (maybe @DouweM too) in Austin to help me testing.

I'm fixing a few things into how Wiki Page events webhook work so we can use it for page updates and remove custom code from Geo. We still need to make a system_hook for it, but most of the code will be based on what I was fixing.

This is the last thing that is currently using the buffered notification.

@brodock thank you for update. wiki via system_hook we can do for 8.9. Make sure for 8.8 we have Single Sign On fixed and benchmarking. Then collaborate with @JobV to announce Geo for 8.8 properly

mentioned in commit 97c33da9

mentioned in commit 031c003a

@brodock status?

Milestone changed to %8.9

Marked the task Geo: Single Sign Out (gitlab-org/gitlab-ee#522) as completed

Marked the task Geo: Benchmark (#560 (closed)) as completed

Geo Single Sign Out is in 8.8, benchmark is done see (#560 (closed)) (jump to the last comment for a summary)

This is nice to have for a GA: "Geo: Wiki sync using system_webhook" but doesn't block anything. License check is disable right now, but re-enabling it to release is a few lines patch (there is code there already)

I was talking to @dzaporozhets and @DouweM about an idea that came after a call with a prospect. Geo currently requires us to elect a primary node where writing operations will happen, all the other nodes will be read-only. This is a requirement right now as setting up a multi-master geographical distributed setup requires a lot of coordination we can't do right now:

Database must be able to handle multi-master updates in a clustered environment (preferably with no global lock, as to prevent latency from killing us)
Git (requires either a global lock mechanism or an intermediary step to merge, elect a winner and rollback others trying to concurrently update a repository, all this 'transactionally')

So an idea is to have a hybrid type of node synchronization, instead of electing a primary, let's say we have 3 regions: A B C, all this regions can do writing operations locally but they may or may not be interested in having access to repositories being worked by the other regions.

Region A, wants region B repositories Region B, wants region A and C repositories Region C, is fine

So you will get in Region A, all their repositories as expected, but also a automatically mirrored (read only) copy of repositories in B.

Some other things that may be good to consider:

How to do authentication?
- We can setup the same OAuth mechanism so you can join from A to B with your current "A" account.
Should we place a link somewhere like: "Go to primary"?
Should we display menu itens for issues/merge request etc that when clicked will bring us back to the primary node for that repository?

cc @JobV

@brodock Thanks for the update. Please create a new issue for the new idea, so it doesn't clutter this discussion :)

@DouweM I've moved the discussion to hybrid synchronization to #623

Thanks @brodock

Marked the task Wiki page events webhook should include Wiki attributes (gitlab-org/gitlab-ce#17507) as completed

A customer ran into an issue trying to set up Geo: https://gitlab.com/gitlab-org/gitlab-ee/issues/628

@gabriel Todos from call earlier this week:

Disable license check in 8-8-stable (!430 (merged))
Improve docs based on notes (!431 (merged))
Extract feature requests and suggested improvements from https://gitlab.zendesk.com/agent/tickets/25447

Todos from https://gitlab.zendesk.com/agent/tickets/25870:

Fix https://gitlab.com/gitlab-org/gitlab-ee/issues/628 in Omnibus
Investigate "invalid text URI" issue from
Investigate "the slave doesn't sync the repos from the master" issue
Investigate "GitLab: API is not accessible " issue
Consider adding a check to ensure that the repos are in sync between the primary and secondary nodes (#727)
Fix OAuth issue with redirect uri (#650 (closed))
Allow to Add custom CA or Trusted self-signed certificate to omnibus environment (omnibus-gitlab#712 (closed))

Other improvements:

Add geo check to rake gitlab:check (Solved partially by doc: gitlab-org/gitlab-ee!431, more here: #727)
Add link to primary in "You can't do this action from secondary" error messages
Add indication to UI that we are on secondary (button in top right with tooltip "Go to primary node") (gitlab-org/gitlab-ee#640)

mentioned in issue #640 (closed)

mentioned in issue #650 (closed)

mentioned in merge request !451 (merged)

Marked the task Geo: The redirect URI included is not valid - OAuth (gitlab-org/gitlab-ee#650) (gitlab-org/gitlab-ee!444) as completed

mentioned in commit 15191864

Marked the task Improve UI for users in a Geo node (gitlab-org/gitlab-ee#640) as completed

mentioned in merge request !459 (merged)

mentioned in commit b594d95d

Marked the task Geo: Wiki sync using system_webhook (Can wait) as completed

Marked the task Geo: Wiki sync using system_webhook (Can wait) as incomplete

Marked the task Geo: Wiki sync using system_webhook (Can wait) as completed

Marked the task Geo: Wiki sync using system_webhook (Can wait) as incomplete

Marked the task Improve gitlab:env:info (gitlab-org/gitlab-ee!459) as completed

Marked the task Omnibus tries to create Postgres extension on read only DB: (gitlab-org/gitlab-ee#628) (omnibus-gitlab!829 (merged)) as completed

Marked the task Omnibus: manage custom SSL certificate (omnibus-gitlab#712 (closed)) as completed

Marked the task Improve required SSH Keys documentation for Geo (!431 (merged)) as completed

Removed ~139607 ~139512 feature proposal labels

Milestone changed to %8.10

Marked the task Geo: Documentation improvements for 8.9 (gitlab-org/gitlab-ee!431) (Can wait) as completed

Added ~360295 label

Added ~139607 ~139512 feature proposal labels

Changed title: GitLab Geo (Read-Only secondary servers) → GitLab Geo (Read-Only secondary servers) (EE Option)

Is there a separate issue for GitLab Geo to address disaster recovery? Like to see Geo address the DR concern. http://docs.gitlab.com/ee/gitlab-geo/disaster-recovery.html. Atlassian Data Center does not support DR for Bitbucket at this time, https://www.atlassian.com/enterprise/data-center

linking to (https://gitlab.com/gitlab-org/gitlab-ee/issues/846)

mentioned in issue #788 (closed)

mentioned in merge request !561 (merged)

mentioned in issue #789 (closed)

mentioned in commit 845f9c31

Marked the task Fix error in admin dashboard when Geo is enabled and current node is nil (#785 (closed)) as completed

Milestone changed to %8.11

Added meta and removed ~360295 ~139512 labels

Removed ~139607 label

Assignee removed

Milestone removed

Milestone changed to %Backlog

mentioned in issue #846

Mentioned in commit pfjason/gitlab-ce@27e2d103

Mentioned in commit pfjason/gitlab-ce@7efdde6b

Mentioned in commit pfjason/gitlab-ce@161a419e

Mentioned in commit pfjason/gitlab-ce@940bcc7b

Mentioned in commit pfjason/gitlab-ce@97e7080d

Mentioned in commit pfjason/gitlab-ce@8281793e

Mentioned in issue #1071 (closed)

Mentioned in issue #1128 (closed)

Mentioned in issue #1129 (closed)

Mentioned in issue #846

Mentioned in issue #1185 (closed)

Mentioned in issue #1190 (closed)

Mentioned in issue omnibus-gitlab#1680 (closed)

Marked the task Geo: more documentation improvements for 8.13 (!766 (merged)) as completed

Mentioned in issue #1217 (closed)

Mentioned in commit faa08fe6

Mentioned in commit 27c85c76

Mentioned in commit b3e45526

Mentioned in commit 9006f046

Mentioned in commit 9284c32a

Mentioned in commit 891b27c0

Mentioned in commit 9c60074d

Mentioned in commit 6c7221b6

Mentioned in commit 2981044f

Mentioned in commit a9e4af35

Mentioned in commit 97c33da9

Mentioned in commit 031c003a

Mentioned in commit 15191864

Mentioned in commit b594d95d

Mentioned in commit 845f9c31

Mentioned in issue #1255

Mentioned in merge request !873 (merged)

Mentioned in merge request !869 (merged)

Mentioned in commit a1a8e297

Marked the task Geo will not let you clone from Secondary on 8.13 (gitlab-org/gitlab-ee#1243) as completed

Mentioned in merge request !875 (merged)

Mentioned in commit 826bcee1

Mentioned in commit 09983fdf

Mentioned in commit 1a78d319

Marked the task Geo: repository is updated but displays old cached data in Web UI (#1129 (closed)) as completed

Mentioned in issue gitlab-com/support#260 (closed)

Mentioned in commit a198d426

Mentioned in commit 1285dbe8

Marked the task Geo: Backfill repositories from primary node without using rsync (#1190 (closed)) as completed

Mentioned in commit maxraab/gitlab-ce@197b1954

Mentioned in commit maxraab/gitlab-ce@c0337aa7

Mentioned in commit maxraab/gitlab-ce@b6608854

Mentioned in merge request !904 (merged)

Mentioned in merge request !905 (merged)

Mentioned in commit 0a807954

Marked the task Geo: Display Custom Avatars (user, project and group) in secondary nodes (#1128 (closed)) as completed

Mentioned in commit 36507aac

mentioned in merge request !952 (merged)

mentioned in merge request omnibus-gitlab!1159 (merged)

mentioned in merge request !978 (merged)

mentioned in commit f8bfa469

marked the task Geo: improve project view UI to guide users how to clone/push from Geo secondary node (#789 (closed)) as completed

mentioned in commit omnibus-gitlab@6d472c51

mentioned in commit omnibus-gitlab@bd284c54

mentioned in commit omnibus-gitlab@0166605e

mentioned in commit e9dda946

marked the task Geo: when license doesn't include Geo you can't disable it anymore (#788 (closed)) as completed

marked the task Ominibus - Geo: Generate SSH keys for gitlab user (omnibus-gitlab#1680 (closed)) as completed

mentioned in issue #1482 (closed)

mentioned in issue #1493 (closed)

mentioned in issue #1520 (closed)

mentioned in issue #1598

mentioned in issue #1611 (closed)

mentioned in issue #1645 (closed)

mentioned in merge request !1157 (merged)

mentioned in merge request !1158 (merged)

mentioned in issue #1366 (closed)

marked the task Geo: Backfill stopped working after 8.15.3 (gitlab-org/gitlab-ee#1645) as completed

mentioned in merge request !998 (merged)

mentioned in issue #1709

mentioned in issue #1744 (closed)

mentioned in merge request !1256 (merged)

mentioned in merge request !1120 (merged)

marked the task Geo: Improve info rake task and create geo specific check task (gitlab-org/gitlab-ee#1611) as completed

mentioned in merge request !1273 (merged)

mentioned in merge request !1348 (merged)

@JobV is there a timeline on when Geo will be GA. Still showing in alpha and many large customers can not use a product that is not GA. In addition, the documentation states that there is a good chance of data loss. Does this issue identify all the tasks needed to move GEO to GA? What is the timing of Geo being GA so we can communicate to prospects and customers?

/cc @sytses

@ChadMalchow we hope to have Geo GA by end of year. We're starting testing Geo on GitLab.com in the coming weeks. Hopefully we'll be able to move it to a Beta status by then, which paves the road to GA.

cc @stanhu

mentioned in commit test-nick/gitlab-ce@ad81f27a

GitLab Geo (Read-Only secondary servers) (EE Option)

Geo decisions

Features

Bugs / improvements

Other ideas / discussions

Designs

Child items ...

Activity

What we've done

Next steps

How we are going to update repositories on secondary servers:

Admin message

Admin message

GitLab Geo (Read-Only secondary servers) (EE Option)

Geo decisions

Features

Bugs / improvements

Other ideas / discussions

Activity

What we've done

Next steps

How we are going to update repositories on secondary servers: