Skip to content

Detect n+1 issues involving Gitaly

Andrew Newdigate requested to merge dev_time_failure_on_haml_to_gitaly into master

What does this MR do?

Counts the number of Gitaly calls that occur for each incoming HTTP request. If the number exceeds 30, an exception is thrown.

There are already multiple places in the code where n + 1 problems are occurring, so the MR also includes a way to make exceptions.

This can be done by wrapping the offending block in a allow_potential_n_plus_1_gitaly_calls_for_block wrapper. Ideally this should be as far down the stack as possible (ie, as near, in stack frames, to the loop that generates the n+1 as possible).

In order to do further analysis on offending routes, the change also adds a gitaly_calls field to lograge: for example

{
  "method": "GET",
  "path": "/moogroup/bootstrap/compare/v3-dev...v4-dev",
  "format": "html",
  "controller": "Projects::CompareController",
  "action": "show",
  "status": 200,
  "duration": 38382.27, 
  "view": 32421.64,
  "db": 17.97,
  "time": "2017-09-05T09:08:43.178Z",
  "params":{
    "namespace_id": "moogroup",
    "project_id": "bootstrap",
    "from": "v3-dev",
    "to": "v4-dev"
  },
  "remote_ip": "127.0.0.1",
  "user_id": 50,
  "username": "moo",
  "gitaly_calls": 783
}

TODO: setup a process for adding new exceptions. Create a new issue for each new exception, mark it as technical debt Gitaly.

TODO: add issues for each exception created in this MR.

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Edited by Andrew Newdigate

Merge request reports