Merge branch '328-versioned-search-ee' into 'master'

Elasticsearch versioned schema for Snippet

See merge request gitlab-org/gitlab-ce!31465

config/initializers/elastic_client_setup.rb

@@ -5,46 +5,42 @@
 require 'gitlab/current_settings'
 Gitlab.ee do
+  require 'elasticsearch/model'
+
+  ### Modified from elasticsearch-model/lib/elasticsearch/model.rb
+
+  [
+    Elasticsearch::Model::Client::ClassMethods,
+    Elasticsearch::Model::Naming::ClassMethods,
+    Elasticsearch::Model::Indexing::ClassMethods,
+    Elasticsearch::Model::Searching::ClassMethods
+  ].each do |mod|
+    Elasticsearch::Model::Proxy::ClassMethodsProxy.include mod
+  end
+
+  [
+    Elasticsearch::Model::Client::InstanceMethods,
+    Elasticsearch::Model::Naming::InstanceMethods,
+    Elasticsearch::Model::Indexing::InstanceMethods,
+    Elasticsearch::Model::Serializing::InstanceMethods
+  ].each do |mod|
+    Elasticsearch::Model::Proxy::InstanceMethodsProxy.include mod
+  end
+
+  Elasticsearch::Model::Proxy::InstanceMethodsProxy.class_eval <<-CODE, __FILE__, __LINE__ + 1
+    def as_indexed_json(options={})
+      target.respond_to?(:as_indexed_json) ? target.__send__(:as_indexed_json, options) : super
+    end
+  CODE
+
+  ### Monkey patches
+
   Elasticsearch::Model::Response::Records.prepend GemExtensions::Elasticsearch::Model::Response::Records
   Elasticsearch::Model::Adapter::Multiple::Records.prepend GemExtensions::Elasticsearch::Model::Adapter::Multiple::Records
   Elasticsearch::Model::Indexing::InstanceMethods.prepend GemExtensions::Elasticsearch::Model::Indexing::InstanceMethods
-
-  module Elasticsearch
-    module Model
-      module Client
-        # This mutex is only used to synchronize *creation* of a new client, so
-        # all including classes can share the same client instance
-        CLIENT_MUTEX = Mutex.new
-
-        cattr_accessor :cached_client
-        cattr_accessor :cached_config
-
-        module ClassMethods
-          # Override the default ::Elasticsearch::Model::Client implementation to
-          # return a client configured from application settings. All including
-          # classes will use the same instance, which is refreshed automatically
-          # if the settings change.
-          #
-          # _client is present to match the arity of the overridden method, where
-          # it is also not used.
-          #
-          # @return [Elasticsearch::Transport::Client]
-          def client(_client = nil)
-            store = ::Elasticsearch::Model::Client
-
-            store::CLIENT_MUTEX.synchronize do
-              config = Gitlab::CurrentSettings.elasticsearch_config
-
-              if store.cached_client.nil? || config != store.cached_config
-                store.cached_client = ::Gitlab::Elastic::Client.build(config)
-                store.cached_config = config
-              end
-            end
-
-            store.cached_client
-          end
-        end
-      end
-    end
-  end
+  Elasticsearch::Model::Adapter::ActiveRecord::Importing.prepend GemExtensions::Elasticsearch::Model::Adapter::ActiveRecord::Importing
+  Elasticsearch::Model::Client::InstanceMethods.prepend GemExtensions::Elasticsearch::Model::Client
+  Elasticsearch::Model::Client::ClassMethods.prepend GemExtensions::Elasticsearch::Model::Client
+  Elasticsearch::Model::ClassMethods.prepend GemExtensions::Elasticsearch::Model::Client
+  Elasticsearch::Model.singleton_class.prepend GemExtensions::Elasticsearch::Model::Client
 end

doc/development/elasticsearch.md

@@ -148,6 +148,36 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/
 - Searches can have their own analyzers. Remember to check when editing analyzers
 - `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches
+## Architecture
+
+GitLab uses `elasticsearch-rails` for handling communication with Elasticsearch server. However, in order to achieve zero-downtime deployment during schema changes, an extra abstraction layer is built to allow:
+
+* Indexing (writes) to multiple indexes, with different mappings
+* Switching to different index for searches (reads) on the fly
+
+Currently we are on the process of migrating models to this new design (e.g. `Snippet`), and it is hardwired to work with a single version for now.
+
+Traditionally, `elasticsearch-rails` provides class and instance level `__elasticsearch__` proxy methods. If you call `Issue.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`, and if you call `Issue.first.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. These proxy objects would talk to Elasticsearch server directly.
+
+In the new design, `__elasticsearch__` instead represents one extra layer of proxy. It would keep multiple versions of the actual proxy objects, and it would forward read and write calls to the proxy of the intended version.
+
+The `elasticsearch-rails`'s way of specifying each model's mappings and other settings is to create a module for the model to include. However in the new design, each model would have its own corresponding subclassed proxy object, where the settings reside in. For example, snippet related setting in the past reside in `SnippetsSearch` module, but in the new design would reside in `SnippetClassProxy` (which is a subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy`). This reduces namespace pollution in model classes.
+
+The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there.
+
+### Creating new version of schema
+
+Currently GitLab would still work with a single version of setting. Once it is implemented, multiple versions of setting can exists in different folders (e.g. `ee/lib/elastic/v12p1` and `ee/lib/elastic/v12p3`). To keep a continuous git history, the latest version lives under the `/latest` folder, but is aliased as the latest version.
+
+If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows:
+
+1. Copy the entire folder of `v12p1` as `v12p3`
+1. Change the namespace for files under `v12p3` folder from `V12p1` to `V12p3` (which are still aliased to `Latest`)
+1. Delete `v12p1` folder
+1. Copy the entire folder of `latest` as `v12p1`
+1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1`
+1. Make changes to `Latest` as needed
+
 ## Troubleshooting
 ### Getting `flood stage disk watermark [95%] exceeded`