Progressively enable elasticsearch for indexed projects

Currently on GitLab.com, we're engaging in backfill to get all our projects indexed. Until that's complete, the ES index we're building is expensive and unused. We can't even evalute its usefulness until the whole process is completed.

For project search, we could switch between ordinary or elasticsearch depending on whether the project has been indexed or not. We have some data for that in the index_status table. All we need to add is tracking for wiki and database indexing (perhaps simple booleans in that table).

Conceivably, we could extend this to group search as well ("use elasticsearch if all projects in this group are indexed, otherwise normal search"), but application search would still need to be manually enabled / disabled.

/cc @smcgivern @vsizov @victorcete

@nick.thomas I think ES search feature can be considered done when it works with zero downtime during servicing. That means that the feature you propose is needed only until ES feature is finished. And one more thing, the global search will not work!

Yep, there's no way to do global search progressively. But we can do the rest, and I do think it would be useful.

Yes, I think so too. Also, we could use Redis to store those flags.

In our discussions about moving database and wiki backfill to background jobs, we've found ourselves in need of these flags too, e.g., in https://gitlab.com/gitlab-org/gitlab-ee/issues/3072

So I think it would be more useful to keep the status persistent, which means not-redis. We've considered putting the status into elasticsearch itself, in the past, which could still be an option, but a first iteration could happily use the existing database table.

I thought we already had an issue for this

I agree that we should do this! cc @victorwu

changed milestone to %Next 2-3 months

@smcgivern we had https://gitlab.com/gitlab-org/gitlab-ee/issues/1607 which is a somewhat different proposition. Is that what you were thinking of?

@nick.thomas no, I think I was just thinking of the discussion at https://gitlab.slack.com/archives/C48D248HG/p1504603399000144. In particular, doing this at the group level would be huge.

mentioned in issue #1186

mentioned in issue gitlab-com/infrastructure#2896 (closed)