Elasticsearch project indexing fails on bad data

While running the backfill rake task on staging, I lose two hours of progress like so:

lupine@worker1:~$ sudo gitlab-rake gitlab:elastic:index_database
I, [2017-03-06T11:49:05.274571 #59129]  INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
Indexing Project records... rake aborted!
NoMethodError: undefined method `+' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:920:in `name_with_namespace'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/elastic/projects_search.rb:49:in `block in as_indexed_json'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/elastic/projects_search.rb:48:in `each'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/elastic/projects_search.rb:48:in `as_indexed_json'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:157:in `as_indexed_json'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:70:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:63:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'

Obviously, we don't want bad data to exist, but we also don't want this process to be quite so fragile. At a minimum, we should catch the error and just skip this instance. Let's try to get that in for 9.0

/cc @vsizov

mentioned in issue #1839 (closed)

nice catch. Yeah, would be nice to handle errors like we do for index_wikis or repository indexer.

One note, we should keep DB records to be processing in batch, like it's now.

Argh, partial backtraces. There's a blank space between the Project.import call happening in elastic.rake and the as_indexed_json method in projects_search.rb.

OK, Project::import comes from Elasticsearch::Model, and we have limited ability to skip over broken records.

I think our best option is to silently swallow this kind of error in as_indexed_json and set the field raising exceptions to a sentinel value like 'INDEXING FAILED'. Wdyt @vsizov?

sounds good.

The sentinel is hard, due to various types. I'll use nil and add to the rails log instead.

mentioned in merge request !1365 (merged)

closed via merge request !1365 (merged)