Elasticsearch database backfill takes too long
In https://gitlab.com/gitlab-org/gitlab-ee/issues/1618 , we moved initial elasticsearch indexing of repositories into sidekiq, as the rake task was taking literally weeks to complete. The database job was "not a bottleneck yet".
I've just had chance to run rake gitlab:elastic:index_database
against staging, and it's taken 30 minutes to index just a quarter of the projects. It hasn't gotten to notes, MRs, issues, etc, yet.
So for GitLab.com, we're looking at more than a working day to run this rake task, which I consider to be unreasonable. We're also missing any sort of status feedback for this part of the indexing process.
/cc @maratkalibek @vsizov