[ES] Note about Index rebuild when update is happening

8.11-ee release requires to rebuild ES indexes just like many other releases and it will be needed in the future releases as well (not every release I hope). We mentioned this in the blog post and update guide, but it seems like many people did not notice that warnings anyway and they have errors because of this. Maybe we need to show the warning when a new omnibus package installed?

@marin @stanhu

Changed title: [ES] Index rebuild during update → [ES] Note about Index rebuild when update is happening

@vsizov How do we rebuild those indexes? We can make this to be automatic on upgrade after the version has changed and if ES is enabled. I am not fully familiar what kind of resources the indexer is using so we will need to check that with the Production team.

I think rebuilding the indexes requires:

1. Deleting the old indexes (making this automatic seems like a bad idea)
2. Calling the Rake task

I think people will definitely miss this. Do we need to have some Elasticsearch index health in the admin page to make this more obvious?

@marin I think making this to be fully automatic is a good idea but we need to ask user permission about that. Because huge projects like GitLab.com is not interested in that because we want to do it by running several tasks in parallel. So we could ask something like "Do you want to rebuild ES indexes? (it can take some time and we recommend to do it manually if your project is big enough) - Yes/No". Is it possible to do it after all services have been restarted and we are already running a new code?

Do we need to have some Elasticsearch index health in the admin page to make this more obvious?

@stanhu I'm not certain about it, but I would not overcomplicate things. You mean something like storing index version in the index and then validate it? and bump every time we need index re-build?

@vsizov No, we don't have an interactive installation and I don't think it is good to introduce it.

I think we might want to do something in between, so I suggest:

1. Create some sort of Elasticsearch health-check on the admin page that would show the status
2. In omnibus-gitlab, create a gitlab-ctl es-index-rebuild that would delete the old indexes and trigger the rake task

I like the idea with gitlab-ctl es-index-rebuild During the update we show information message now (about reconfogure command and so on), so we can include note about ES as well.

@marin would it be hard to enable/disable that message for certain release? Maybe some configuration file in repository or something?

Create some sort of Elasticsearch health-check on the admin page that would show the status

Right, that's what I was thinking. In this case, if the new indexes don't even exist, we should mention that and link to documentation on how to fix it.

Right, that's what I was thinking.

Sorry, I should have said Create some sort of Elasticsearch health-check on the admin page that would show the status, as proposed by Stan :)

would it be hard to enable/disable that message for certain release?

@vsizov We don't have a way of doing that at the moment so I don't really know. I would think that just having hardcoded message now is better than none, we can think of improving it as we go.

One more confusion https://gitlab.com/gitlab-org/gitlab-ee/issues/1046 We need to implement it ASAP

Mentioned in issue #1046 (closed)

I think health-check won't prevent the situation like linked one above ^ I think health check is a good one but this is different story, now we need something like "pending migration warning"

@vsizov Do we already have the rake task to do reindex? If yes then we need an issue in omnibus-gitlab so we can schedule it.

Milestone changed to %8.13

Moved to omnibus-gitlab#1608