Instrument ES indexer
Activity
@yorickpeterse I would love to instrument the ES indexer https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer The main reason to create this file was to avoid spawning whole rails application there. For now I see that I need to load lots of stuff to instrument this Repository class, maybe you have a better idea?
/cc @pcarranza
@vsizov Instrumentation indeed requires the whole Rails application, not just for the source code but also for the InfluxDB settings. Can we perhaps turn the indexer into a periodically executing Sidekiq worker? In that case the worker will be instrumented automatically and there's nothing extra we'd need to do.
-
@yorickpeterse Actually it's already a Sidekiq job I mean Sidekiq job runs the indexer as a separate process and wait the results. We have done it in this way because of lots of memory leaks in indexer. BUT if we wait the indexer we can probably just use Sidekiq instrumentation just like you said because we don't need too many details like how much does it take to index commits or blobs. So thanks for the idea.
UPDATE: At least we can see timings
Edited by Valery Sizov -
@pcarranza Your original request was to use logging which make a lot of sense to me. So I will work on this direction as well.
Thanks @vsizov I think that we need to break the blackbox that elasticsearch execution is in way that we can track what's it doing at a given time - what's being indexed, how much time it takes, did it failed? if so, why? etc.
-
It's not a blackbox, we catch errors and re-raising them so they are handled properly https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/elastic/indexer.rb#L36
When it fails the sidekiq fails as well just like any other part of the project.
Edited by Valery Sizov mentioned in issue gitlab-com/infrastructure#355 (closed)
mentioned in merge request !683 (merged)
Status changed to closed by merge request !683 (merged)
