Move from short-lived indexer process to long-lived indexing daemon

In the current (9.0, 9.1) architecture, multiple instances of bin/elastic_repo_indexer / es-git-go are spawned in parallel by Sidekiq - one per git push. We get very little control over priority, concurrency or IO load as a result.

By converting es-git-go into a long-lived daemon that runs on N hosts, we can pass it notifications when a git push happens, and make intelligent scheduling decisions to limit the total CPU, RAM and IOPS load of elasticsearch indexing.

I propose doing this for v9.2 - an initial implementation can introduce an es-git-go serve mode, and have es-git-go <project_id> <path> perform the enqueuing. Responsibility for updating the index_statuses table would move from gitlab-ee to es-git-go.

/cc @vsizov @stanhu

@nick.thomas I think this is generally a good idea. As an option, we could use Redis' subscribe mechanism like Sidekiq does.

Responsibility for updating the index_statuses table would move from gitlab-ee to es-git-go.

I think we should revise needing of this table at all. Because it becomes painful in this case.

@nick.thomas Also note that we can throttle Sidekiq queues, so it's not like "totally uncontrolled" now but having it as a separate service will give us some flexibility...

How would es-git-go get the database connection/credentials for the index_statuses table? Another config file? Or through an API with unicorn? Things get a little complicated if there are TLS certificates involved too. I'd be concerned that Rails would have an ORM that was quietly manipulated by this process.

@stanhu it's a table that has essentially no connection with the rails application - it's used solely by indexing, to track indexing status - so using database integration here isn't particularly worrying to me.

As an alternative, we could store the indexing status in elastisearch itself, and query that instead of postgresql.

mentioned in merge request !1 (merged)

I've been thinking about how to get the git push notifications in a long-lived process model. Currently, we enqueue a sidekiq job per git push, and this supervises the indexer process to completion.

Easiest is to modify the gitlab-repository-indexer so that when it's invoked, it just pushes ELASTIC_CONNECTION_INFO, RAILS_ENV, and the project ID and path, to the persistent process via an internal mechanism.

Slightly harder is to replace that sidekiq job with a similar (but simpler) pubsub mechanism to the solution for "CI long polling".

Much harder, but perhaps more useful in the long term, is to add an explicit event bus and emit "push event" notifications on it.

The main benefits of this issue come from being able to more effectively throttle the amount of capacity we give to elasticsearch. Given the outcome of https://gitlab.com/gitlab-com/infrastructure/issues/1157, I don't believe this is a priority so I'm happy not addressing in 9.2. I'll leave the issue open for now, though.

More discussion about a long-lived indexer: https://gitlab.com/gitlab-com/infrastructure/issues/2449#note_37069629

We could use redis to receive data to be indexed from gitlab-ce.