Migrate to Elasticsearch 5.1.* when it's stable enough

• Remove requirement for delete_by_query plugin as it's already built-in
• Make sure search works as expected (changes to the mapping and some search queries are needed here)
• Make sure documents gets deleted when project is deleted (API to delete_by_query functionality has been changed)
• Update rake gitlab:app:check task
• Requirements
• Update the update guide
• Check if we really need to rebuild index. Maybe that's enough to migrate index with ES official migrator.

References: https://gitlab.com/gitlab-org/gitlab-ee/issues/1248 https://gitlab.com/gitlab-org/gitlab-ee/issues/1254 https://gitlab.com/gitlab-org/gitlab-ee/issues/1251

Mentioned in issue #1248 (closed)

mentioned in issue gitlab-com/infrastructure#388 (closed)

mentioned in issue #1503 (closed)

assigned to @vsizov

Can't get highlighting work on 5.1. Trying to understand what's wrong with our query.

Search is already working locally on 5.1 Now I'm solving the issue with delete-by-query plugin because now it's built-in but elasticsearch-ruby official gem does nor support it directly yet, so I need to find some workaround.

I just fixed data wiping when project is removed. Next action is to fix specs related to ES 5.x migration

WIP MRs https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1059 and https://gitlab.com/gitlab-org/gitlab-elasticsearch-git/merge_requests/24

changed title from Migrate to Elasticsearch 5.0. when it's stable enough* to Migrate to Elasticsearch 5.1. when it's stable enough*

I released gem gitlab-elasticsearch-git 1.1.0 and created an issues about absorbing it to EE https://gitlab.com/gitlab-org/gitlab-elasticsearch-git/issues/6

marked the task Remove requirement for delete_by_query plugin as it's already built-in as completed

marked the task Make sure everything works as expected as completed

Everything works now. I need to change the update guide yet and check how smoothy everything works when we update index from 2.4.x to 5.1.x. The ES official guide says that indexes are compatible but I would rather check myself just in case.

marked the task Update the update guide as completed

marked the task Check if we really need to rebuild index. Maybe that's enough to migrate index with ES official migrator. as completed

I created an issue about update guide https://gitlab.com/gitlab-org/gitlab-ee/issues/1585 and put it to my personal TODO. The text is pretty short. Also I tested update ES from 2.4.x to 5.1.0. It works great except one confusion in their documentation, I created the issue https://github.com/elastic/elasticsearch/issues/22744

mentioned in merge request !1059 (merged)

changed milestone to %9.0

closed via merge request !1059 (merged)