ElasticSearch initial index: Make rake tasks complete in a reasonable time

In discussion with @pcarranza , he noted that elasticsearch initial indexing is kicked off using a rake tasks. The length of time this task takes to complete depends on the number of items to index, and in the case of GitLab.com, it can take several weeks. Multiple rake processes are started

He proposed re-implementing the rake task in terms of sidekiq jobs - rather than synchronously indexing everything, it can just enqueue a job for each outstanding piece of work. These can run in a dedicated sidekiq queue and have their concurrency controlled using the normal sidekiq mechanisms, rather than spinning up rake processes in an ad-hoc manner.

/cc @maratkalibek @vsizov

added ~147208 label

Sounds great in theory but I don't think it will work practically. Our Sidekiq job are not reliable, they can be killed at any given time. Sidekiq jobs are great at performing single, atomic job. I hardly imagine how to enqueue several millions of jobs for every project on GitLab.com.

But I understand the problem, it's really painful to run that rake task manually and several times. We should brainstorm it.

We discussed a dedicated indexer process a little in #1606 (closed) - this could conceivably take on initial indexing as well as index updates, but not in time for 8.17.

I'm going to do some sanity testing of the "create a sidekiq job for each of the things" approach; it doesn't strike me as being immediately beyond redis's capabilities.

mentioned in merge request !1143 (merged)

On the other hand, if we pick the "continue from last checkpoint" feature from our rake task it does not seem like insane to me. We could create several Sidekiq jobs and assign the range of project IDs to each. If some job fail it will be restarted and it will continue from the last checkpoint. It's also possible to create the dedicated Web page for indexer where you can start indexer and check the status of indexing.

Yes, a job per batch might be more sensible. I experimented a little with job per project, and just inserting jobs into Sidekiq with push_bulk was taking > 40 minutes!

We don't currently keep the index_statuses table up to date during normal indexer operation - should we?

The "project ID range" stuff is currently focused on the repositories. How about the DB tables? Do those rake tasks complete quickly enough on GitLab.com that we don't need to do anything special for them?

mentioned in merge request !1144 (merged)

We don't currently keep the index_statuses table up to date during normal indexer operation - should we?

It's intended to work with rake task only. But If we move indexer entirely to Sidekiq we would need to implement it.

How about the DB tables? Do those rake tasks complete quickly enough on GitLab.com that we don't need to do anything special for them?

AFAIK It was not a bottleneck so far but I think if we move indexer to Sidekiq we could index DB with repository in the same job.

closed via merge request !1144 (merged)

reopened

removed milestone

changed milestone to %8.17

closed