[ES] Note about Index rebuild when update is happening

8.11-ee release requires to rebuild ES indexes just like many other releases and it will be needed in the future releases as well (not every release I hope). We mentioned this in the blog post and update guide, but it seems like many people did not notice that warnings anyway and they have errors because of this. Maybe we need to show the warning when a new omnibus package installed?

@marin @stanhu

Changed title: [ES] Index rebuild during update → [ES] Note about Index rebuild when update is happening

@vsizov How do we rebuild those indexes? We can make this to be automatic on upgrade after the version has changed and if ES is enabled. I am not fully familiar what kind of resources the indexer is using so we will need to check that with the Production team.

I think rebuilding the indexes requires:

1. Deleting the old indexes (making this automatic seems like a bad idea)
2. Calling the Rake task

I think people will definitely miss this. Do we need to have some Elasticsearch index health in the admin page to make this more obvious?

@marin I think making this to be fully automatic is a good idea but we need to ask user permission about that. Because huge projects like GitLab.com is not interested in that because we want to do it by running several tasks in parallel. So we could ask something like "Do you want to rebuild ES indexes? (it can take some time and we recommend to do it manually if your project is big enough) - Yes/No". Is it possible to do it after all services have been restarted and we are already running a new code?

Do we need to have some Elasticsearch index health in the admin page to make this more obvious?

@stanhu I'm not certain about it, but I would not overcomplicate things. You mean something like storing index version in the index and then validate it? and bump every time we need index re-build?

@vsizov No, we don't have an interactive installation and I don't think it is good to introduce it.

I think we might want to do something in between, so I suggest:

1. Create some sort of Elasticsearch health-check on the admin page that would show the status
2. In omnibus-gitlab, create a gitlab-ctl es-index-rebuild that would delete the old indexes and trigger the rake task

I like the idea with gitlab-ctl es-index-rebuild During the update we show information message now (about reconfogure command and so on), so we can include note about ES as well.

@marin would it be hard to enable/disable that message for certain release? Maybe some configuration file in repository or something?

Create some sort of Elasticsearch health-check on the admin page that would show the status

Right, that's what I was thinking. In this case, if the new indexes don't even exist, we should mention that and link to documentation on how to fix it.

Right, that's what I was thinking.

Sorry, I should have said Create some sort of Elasticsearch health-check on the admin page that would show the status, as proposed by Stan :)

would it be hard to enable/disable that message for certain release?

@vsizov We don't have a way of doing that at the moment so I don't really know. I would think that just having hardcoded message now is better than none, we can think of improving it as we go.

One more confusion https://gitlab.com/gitlab-org/gitlab-ee/issues/1046 We need to implement it ASAP

I think health-check won't prevent the situation like linked one above ^ I think health check is a good one but this is different story, now we need something like "pending migration warning"

@vsizov Do we already have the rake task to do reindex? If yes then we need an issue in omnibus-gitlab so we can schedule it.

Milestone changed to gitlab-org/gitlab-ee%17

Moved from gitlab-org/gitlab-ee#922

The rake task is already exist gitlab:elastic:index but it's still not clear what to do here.

@WarheadsSE One other cool thing to work on as it is simple to implement. Ping me and/or @twk3 for help.

Added EE label

At this point, are we still sorting out how to define when this message should or should not be shown? The task is present, but should we simply be telling people to run it on upgrade along with reconfigure, or should be be using a state/flag for denoting that it will be needed?

If we're flagging, how do we know the different between the prior installed version and the upcoming version? Should we care other than having some simple map that says what versions require the rebuild, and a state of when the last rebuild was done?

@WarheadsSE Smallest step possible is to implement the gitlab-ctl wrapper around the rake task. We will then tell users to trigger this command in the release post. After we ship that, we can start thinking about the flag/automatic check, this will require some work with the backend team.

@marin I agree with that process. Just thinking down the line, knowing that our first steps have to be incremental.

Milestone changed to %8.15

Added ~113496 label

changed milestone to %8.16

changed milestone to %9.0

@balasankarc Could you look into this when you get some time? It is not urgent. We need to check whether the issue is still valid given the re-writes that ES support received in the mean time.

changed milestone to %Backlog

/cc @nick.thomas

Given the various ad-hoc changes we make to the elasticsearch indexes from time to time, I think it would be useful to have somethine like the schema_migrations table in AR. The index schema could be managed with migrations that happen either with or without downtime, as we do for the SQL side of things.

It's quite a bit of tooling work, but would be a lot more robust than what we have at the moment, and would allow us to automatically update the ES schema without user intervention.

Rebuild-the-world scenarios are awful and should be avoided whenever possible.

@nick.thomas Yeah, but unfortunately, you need to rebuild the index if you change anything. The only exception is when you add a new field.

@nick.thomas So, will this be happening at GitLab-CE codebase or is a rake task (or a gitlab-ctl command) in omnibus-gitlab still necessary? From your comment above, I assume the former.

@balasankarc it's an EE-only thing. Currently, we introduce an ad-hoc rake task and give instructions on running it manually in the release notes (or say to rebuild the index if that's needed).

The aspiration is to move to something that doesn't need manual intervention when upgrading.

added EE Starter label