Geo will not let you clone from Secondary on 8.13

Summary

On 8.13.<=5, cloning a repo from a Secondary node to a local machine does not work.

What I think this is

This migration: db/migrate/20161007073613_create_user_activities.rb Adds the user_activities table.

This error seems to be, in part, the result of trying to update the last_activity_at field in the user_activities table in the read only databasy of the secondary server, when a user tries to clone a repo.

Steps to reproduce

1. Set up GitLab Geo with 8.13.5 and create a project with a file in the repo.
2. Use the Clone URL from the secondary node to clone to your local machine.

Expected behavior

It should clone the repo.

Actual behavior

➜  test ls
➜  test git clone git@secondary:dewet/test-push.git
Cloning into 'test-push'...
GitLab: API is not accessible
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

In the following, I successfully clone from the primary, right after cloning from the secondary fails:

➜  test git clone git@primary:dewet/test-push.git
Cloning into 'test-push'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
➜  test ls
test-push
➜  test

Relevant logs and/or screenshots

ActiveRecord::StatementInvalid (PG::ReadOnlySqlTransaction: ERROR:  cannot execute UPDATE in a read-only transaction
: UPDATE "user_activities" SET "last_activity_at" = '2016-11-14 08:47:22.407725' WHERE "user_activities"."id" = $1):
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `exec_prepared'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_cache'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:173:in `exec_delete'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:114:in `update'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:345:in `update_all'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:299:in `update_columns'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:273:in `update_column'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/user_activity.rb:13:in `touch'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/users/activity_service.rb:17:in `record_activity'
  /opt/gitlab/embedded/service/gitlab-rails/app/services/users/activity_service.rb:11:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/internal.rb:52:in `log_user_activity'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/internal.rb:80:in `block (2 levels) in <class:Internal>'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:66:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:66:in `block (2 levels) in generate_api_method'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:65:in `block in generate_api_method'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:249:in `block in run'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:230:in `run'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:300:in `block in build_stack'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/base.rb:30:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/base.rb:23:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/base.rb:30:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/base.rb:23:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/error.rb:29:in `block in call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/error.rb:28:in `catch'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/error.rb:28:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/middleware/base.rb:23:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:214:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/endpoint.rb:208:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:152:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:96:in `block in recognize'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:68:in `optimized_each'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-mount-0.8.3/lib/rack/mount/code_generation.rb:95:in `recognize'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-mount-0.8.3/lib/rack/mount/route_set.rb:141:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/api.rb:101:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/api.rb:43:in `call!'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/grape-0.15.0/lib/grape/api.rb:38:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/mapper.rb:51:in `serve'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `each'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/journey/router.rb:30:in `serve'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/routing/route_set.rb:817:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:15:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:16:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:35:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `catch'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-cors-0.4.0/lib/rack/cors.rb:80:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-attack-4.3.1/lib/rack/attack.rb:106:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/etag.rb:24:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:38:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/readonly_geo.rb:29:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/flash.rb:260:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/query_cache.rb:36:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:38:in `call_app'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:20:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:68:in `block in tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:26:in `tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/tagged_logging.rb:68:in `tagged'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/rack/logger.rb:20:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/actionpack-4.2.7.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/lock.rb:17:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sentry-raven-2.0.2/lib/raven/integrations/rack.rb:50:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/engine.rb:518:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/application.rb:165:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `public_send'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `method_missing'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:132:in `start'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/bin/unicorn:22:in `load'
  /opt/gitlab/embedded/service/gem/ruby/2.3.0/bin/unicorn:22:in `<top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.6/lib/bundler/cli/exec.rb:74:in `load'
  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.6/lib/bundler/cli/exec.rb:74:in `kernel_load'
  /opt/gitlab/embedded/lib/ruby/g

Output of checks

dewet@secondary:~$ sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect"
[sudo] password for dewet:
#<GeoNode id: 2, schema: "http", host: "secondary", port: 80, relative_url_root: "", primary: false, geo_node_key_id: 2, oauth_application_id: 1, system_hook_id: 2>

Confirmed that this is a regression

I uninstalled GitLab 8.13.5 and installed 8.12.9 on both instances and the problem went away, I can now clone from Secondary on the 8.12.9 setup.

This is the second Geo regression like this that I'm aware of - !834 (merged) has definite echoes.

Do we need a section in the database styleguide or MR performance guidelines saying "beware updating the database in a nominal read" ?

@dewetblomerus This is caused by https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/api/internal.rb#L57. /cc @jameslopez

Hmm I wonder if we can centralise that somewhere so we don't forget about this - return if Gitlab::Geo.secondary?

@rymai to be more precise this is the issue:

https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/services/users/activity_service.rb#L17

@jameslopez I don't think we can do that right now, the places where we "patch" for Geo are very unrelated to each other. What we can do in the future is have interchangeable implementations for critical parts that Geo depends on, so instead of

::Users::ActivityService.new(...)

We would have something like:

::Users.activity_service.record(...)

def self.activity_service
   if Gitlab::Geo.secondary?
     ::Users::LogActivityService.new
   else
     ::Users::DbActivityService.new
   end 
end

We also have to start thinking how we can deal with data that the secondary "has" to keep (like audit logs etc).

Added ~164929 label

Mentioned in merge request !873 (merged)

Mentioned in issue #76

Status changed to closed by merge request !873 (merged)

Milestone changed to %8.13

Fixed by !873 (merged), should be into next patch release. cc @eReGeBe

@brodock FYI I'm still the release manager for %8.13. ;) Thanks for the fix!

Mentioned in commit 826bcee1

marked this issue as related to #1744 (closed)