`gitlab:env:info` rake task crashes in Geo secondary because of `PG::ReadOnlySqlTransaction` exception
Summary
Running rake gitlab:env:info (or gitlab-rake equivalent) crashes in Geo (#76) secondary (see attached stacktrace below)
Steps to reproduce
Go to a Geo secondary node and try to run gitlab:env:info rake task.
Relevant logs and/or screenshots
git@gabriel-geo:~/gitlab$ rake gitlab:env:info
WARNING: This version of GitLab depends on gitlab-shell 4.1.1, but you're running 3.6.4. Please update gitlab-shell.
System information
System: Ubuntu 14.04
Current User: git
Using RVM: yes
RVM Version: 1.28.0
Ruby Version: 2.3.1p112
Gem Version: 2.6.8
Bundler Version:1.13.6
Rake Version: 10.5.0
Sidekiq Version:4.2.7
rake aborted!
ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction
: INSERT INTO "project_features" ("created_at", "updated_at", "builds_access_level", "issues_access_level", "merge_requests_access_level", "snippets_access_level", "wiki_access_le
vel", "repository_access_level") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id"
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `exec_prepared'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_cache'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:192:in `exec_insert'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in `insert'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/relation.rb:64:in `insert'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:524:in `_create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/counter_cache.rb:139:in `_create_record'
/home/git/gitlab/config/initializers/ar_monkey_patch.rb:12:in `_create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/attribute_methods/dirty.rb:133:in `_create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/callbacks.rb:306:in `block in _create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_create_callbacks'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/callbacks.rb:306:in `_create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/timestamp.rb:57:in `_create_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:504:in `create_or_update'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/callbacks.rb:302:in `block in create_or_update'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:117:in `call'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:505:in `call'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/callbacks.rb:302:in `create_or_update'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:120:in `save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/validations.rb:37:in `save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/attribute_methods/dirty.rb:21:in `save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:286:in `block (2 levels) in save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:220:in `transaction'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:286:in `block in save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/transactions.rb:285:in `save'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/persistence.rb:241:in `update_attribute'
/home/git/gitlab/app/models/concerns/project_features_compatibility.rb:35:in `write_feature_attribute'
/home/git/gitlab/app/models/concerns/project_features_compatibility.rb:22:in `issues_enabled='
/usr/share/rvm/gems/ruby-2.3.1/gems/default_value_for-3.0.2/lib/default_value_for.rb:178:in `block in set_default_values'
/usr/share/rvm/gems/ruby-2.3.1/gems/default_value_for-3.0.2/lib/default_value_for.rb:154:in `each'
/usr/share/rvm/gems/ruby-2.3.1/gems/default_value_for-3.0.2/lib/default_value_for.rb:154:in `set_default_values'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:432:in `block in make_lambda'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:228:in `block in halting_and_conditional'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in `block in call'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in `each'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in `call'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/usr/share/rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in `_run_initialize_callbacks'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/core.rb:284:in `initialize'
/usr/share/rvm/gems/ruby-2.3.1/gems/default_value_for-3.0.2/lib/default_value_for.rb:142:in `initialize'
/usr/share/rvm/gems/ruby-2.3.1/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:463:in `initialize'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/reflection.rb:141:in `build_association'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:250:in `build_record'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_association.rb:146:in `build'
/usr/share/rvm/gems/ruby-2.3.1/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_proxy.rb:259:in `build'
/home/git/gitlab/lib/tasks/gitlab/info.rake:34:in `block (3 levels) in <top (required)>'
/usr/share/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `eval'
/usr/share/rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `<main>'Possible fixes
The issue is in https://gitlab.com/gitlab-org/gitlab-ee/blob/0515a4470c2a9419929be7d3a7a9279485e0598d/app/models/concerns/project_features_compatibility.rb#L35
If we change that line to :
project_feature.send(:write_attribute, field, access_level)We don't try to update a table column when we are just using .build_*. This may also improve specs speed as we are not going to do 2 database write operations whenever we create a project.
I'm not sure what are the implications of doing that and why it was node in that way. Should we create a migration to "fix" existing data? Should we schedule async tasks?
cc @DouweM
