Undefined method `quote' for class `Gitlab::Database::LoadBalancing::ConnectionProxy

I saw this while hunting for Error 500s in the logs:

NameError (undefined method `quote' for class `Gitlab::Database::LoadBalancing::ConnectionProxy'):
  app/models/ee/project.rb:247:in `method'
  app/models/ee/project.rb:247:in `secret_variables_for'
  app/models/ci/build.rb:388:in `secret_variables'
  app/models/ci/build.rb:208:in `variables'
  lib/gitlab/metrics/instrumentation.rb:156:in `block in variables'
  lib/gitlab/metrics/method_call.rb:23:in `measure'
  lib/gitlab/metrics/instrumentation.rb:156:in `variables'
  lib/api/helpers.rb:261:in `handle_api_exception'
  lib/gitlab/metrics/instrumentation.rb:156:in `block in handle_api_exception'
  lib/gitlab/metrics/method_call.rb:23:in `measure'
  lib/gitlab/metrics/instrumentation.rb:156:in `handle_api_exception'
  lib/api/api.rb:81:in `block in <class:API>'
  lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/database/load_balancing/rack_middleware.rb:37:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/rails_queue_duration.rb:20:in `call'
  lib/gitlab/metrics/rack_middleware.rb:29:in `block in call'
  lib/gitlab/metrics/transaction.rb:49:in `run'
  lib/gitlab/metrics/rack_middleware.rb:29:in `call'

I don't see it in Sentry, but I thought it was an unusual message.

/cc: @godfat, @ayufan

added regression label

changed milestone to %9.4

mentioned in issue #35092 (closed)

assigned to @godfat

marked this issue as related to #35111 (closed)

Can someone figure out why this exception didn't show up in Sentry?

Reopen to track why this wasn't showing up in Sentry

reopened

It could be that NameError isn't a status 500 error: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/api/helpers.rb#L431

@stanhu That doesn't make sense because NameError shouldn't be responding to status:

>> begin; 0.method(:foo); rescue => e; p e.respond_to?(:status); end
false
=> false

And it should be reported if it's not responding to status.

Right. I thought it might have been the Sentry library ignoring this error, but it is not in the list of excluded errors: https://github.com/getsentry/raven-ruby/blob/master/lib/raven/configuration.rb#L152-L161

Looking at the backtrace again, I don't really understand why variables was called, particularly:

  lib/gitlab/metrics/instrumentation.rb:156:in `variables'
  lib/api/helpers.rb:261:in `handle_api_exception'

It looks to me like the exception was raised duration the process of reporting another exception... but I don't see why it's trying to capture variables?

I'm guessing that is Sentry gathering the state of variables, secret variables, etc.

Yeah, but I don't see where it's trying to fetch that. I am guessing that it might be trying to gather all the instance variables, which could be @build, but then it shouldn't really be calling @build.variables because it's not used in @build.inspect nor @build.attributes.

@godfat What is the status of that?

@ayufan The bug was fixed in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2430 but we don't know why it's not showing up in Sentry yet.

I still see some failures on CI API v1. Can this be the same problem? It is not happening on CE, only EE.

However, no Sentry log ;(

@ayufan Do you know if that patch was applied there? Also I don't get why Sentry won't log this. Were there more logs I could see?

Yes, it was applied as this is needed to have it working in general, but something is still not right on CI API v1.

@godfat Any ideas why Sentry is not capturing these exceptions? I think @ayufan had a similar case where this happened.

@stanhu No idea :( It would be nice if we have more logs or data to investigate. On the other hand, where did you find the logs? Could it be that those nodes don't have Sentry enabled? (e.g. api worker?)

What puzzled me more is, @ayufan said that this happened after the patch. That shouldn't happen... as searching against method(:quote) or method :quote doesn't give any results.

@godfat I didn't find the logs in Sentry, so I went digging on our log aggregator to find 500 errors. I didn't see this problem happen after the patch, but I know @ayufan found https://gitlab.com/gitlab-org/gitlab-ce/issues/35421 the same way. It sounds like you should be able to replicate that problem easily.

mentioned in issue #35979

@stanhu Ohhh.. so I might misunderstand. This should be fixed however Sentry wasn't capturing some exceptions are a more general issue :( I thought it's only for this one.

Let me create a new issue then: https://gitlab.com/gitlab-org/gitlab-ce/issues/35979

closed

marked this issue as related to #35979

removed the relation with #35979

marked this issue as related to #35979

mentioned in merge request !14785