Elasticsearch indexer attempts to reindex entire repository with pull mirror
If you set a project as a pull mirror, ElasticCommitIndexerWorker will be run without a from_sha and a to_sha, causing a full reindex of a repo: https://gitlab.com/gitlab-org/gitlab-ee/blob/v9.3.5-ee/app/models/ee/project/import_status_state_machine.rb#L50
For large repositories like the Linux repository, this can bring the system to its knees.
Activity
/cc: @nick.thomas, @vsizov
@stanhu good catch!
We store the last successfully indexed SHA in the database, so this is quite easy to fix, at least. I'll put together an MR.
assigned to @nick.thomas
Incidentally, the (v4.4) Linux kernel was my test case during indexing. It would take ~40 minutes to be completely indexed with the Ruby indexer, which was reduced to ~10 with the experimental one: https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/issues/1#note_26153608
mentioned in merge request !2393 (merged)
changed milestone to %9.4
closed via merge request !2393 (merged)
mentioned in commit 6d229e8a
mentioned in commit ed31a078
