Codeclimate widget struggles with big codelimate.json files

We got a quiet big project on our GitLab instance and tried the new codeclimate feature. After running codeclimate for the first time it generated a 25MB codeclimate.json file.

If we now open a new merge request, the browser gets unresponsive after a few seconds while trying to diff both files (I guess).

As customer I would assume that the codeclimate feature also can handle/analyze bigger files.

changed the description

changed the description

changed title from Codeclimate struggles with big files to Codeclimate widget struggles with big codelimate.json files

mentioned in issue #2731 (closed)

Another customer has run into this issue: https://gitlab.zendesk.com/agent/tickets/79277.

/cc @victorwu

added ~155960 ~874214 code review customer labels

We have the same problem with CE project on .com as we load on a simple MR 2 times a 2.1 MB big file :

https://gitlab.com/gitlab-org/gitlab-ce/issues/33719#note_33554520

Is it possible to generate stripped codeclimate.json that would only contain metrics and offenses?

cc @dzaporozhets

while "doing the diff inside GL" might help a few, I´d suspect another issue elsewhere...

• 2x45MB ticks to 1.6gb max_mem and goes unresponsive for me
• 2x15MB doesnt get to load for me ether
• 2x25MB goes unresponsive for the reporter of this issue
• 2x2MB even gets reported as issue /via @timzallmann https://gitlab.com/gitlab-org/gitlab-ee/issues/2737#note_33554868
• 2x2.5MB "works for me" here (random example https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2244/pipelines ) - but still creates a 40MB codeclimateMetrics-JS object for it

Is it possible to generate stripped codeclimate.json that would only contain metrics and offenses?

@ayufan looking at json it actually contains only metrcis and offenses. I believe the content -> body section eats a lof of space. Considering huge amount of issues in the project this repeating text can be 50% of file size easily. See example:

[  
  {  
    "type":"Issue",
    "check_name":"file_access",
    "description":"Model attribute used in file name",
    "fingerprint":"87526ba3cf2b7e2ee9f0f0ea555c84ae4031b353fd8ced591d7d7ff907a4e1eb",
    "categories":[  
      "Security"
    ],
    "severity":"normal",
    "remediation_points":300000,
    "location":{  
      "path":"lib/backup/repository.rb",
      "lines":{  
        "begin":127,
        "end":127
      }
    },
    "content":{  
      "body":"Using user input when accessing files (local or remote) will raise a warning in Brakeman.\n\nFor example\n\n    File.open(\"/tmp/#{cookie[:file]}\")\n\nwill raise an error like\n\n    Cookie value used in file name near line 4: File.open(\"/tmp/#{cookie[:file]}\")\n\nThis type of vulnerability can be used to access arbitrary files on a server (including `/etc/passwd`.\n"
    },
    "engine_name":"brakeman"
  },
  {  
    "type":"Issue",
    "check_name":"regex_dos",
    "description":"Model attribute used in regex",
    "fingerprint":"b0f801806cb42e892d9dd36783914d6d1b2f4d1faf4a3df3e5688186e655f91c",
    "categories":[  
      "Security"
    ],
    "severity":"normal",
    "remediation_points":300000,
    "location":{  
      "path":"app/models/commit.rb",
      "lines":{  
        "begin":91,
        "end":91
      }
    },
    "content":{  
      "body":"Denial of Service (DoS) is any attack which causes a service to become unavailable for legitimate clients.\n\nFor issues that Brakeman detects, this typically arises in the form of memory leaks.\n\n### Symbol DoS\n\nSince Symbols are not garbage collected in Ruby versions prior to 2.2.0, creation of large numbers of Symbols could lead to a server running out of memory.\n\nBrakeman checks for instances of user input which is converted to a Symbol. When this is not restricted, an attacker could create an unlimited number of Symbols.\n\nThe best approach is to simply never convert user-controlled input to a Symbol. If this cannot be avoided, use a whitelist of acceptable values.\n\nFor example:\n\n    valid_values = [\"valid\", \"values\", \"here\"]\n\n    if valid_values.include? params[:value]\n      symbolized = params[:value].to_sym\n    end\n\n\n### Regex DoS\n\nRegular expressions can be used for DoS if the pattern and input requires exponential time to process.\n\nBrakeman will warn about dynamic regular expressions which may be controlled by an attacker. The attacker can create an \"[evil regex](https://www.owasp.org/index.php/Regular_expression_Denial_of_Service_-_ReDoS)\" and then supply input which causes the server to use a large amount of resources.\n\nIt is recommended to avoid interpolating user input into regular expressions.\n"
    },
    "engine_name":"brakeman"
  },
  {  
    "type":"Issue",
    "check_name":"regex_dos",
    "description":"Model attribute used in regex",
    "fingerprint":"3ea396be233ceab55e796919e67b801c058713ba69d8d24151df13b14ab2d5fc",
    "categories":[  
      "Security"
    ],
    "severity":"normal",
    "remediation_points":300000,
    "location":{  
      "path":"app/models/commit_range.rb",
      "lines":{  
        "begin":45,
        "end":45
      }
    },
    "content":{  
      "body":"Denial of Service (DoS) is any attack which causes a service to become unavailable for legitimate clients.\n\nFor issues that Brakeman detects, this typically arises in the form of memory leaks.\n\n### Symbol DoS\n\nSince Symbols are not garbage collected in Ruby versions prior to 2.2.0, creation of large numbers of Symbols could lead to a server running out of memory.\n\nBrakeman checks for instances of user input which is converted to a Symbol. When this is not restricted, an attacker could create an unlimited number of Symbols.\n\nThe best approach is to simply never convert user-controlled input to a Symbol. If this cannot be avoided, use a whitelist of acceptable values.\n\nFor example:\n\n    valid_values = [\"valid\", \"values\", \"here\"]\n\n    if valid_values.include? params[:value]\n      symbolized = params[:value].to_sym\n    end\n\n\n### Regex DoS\n\nRegular expressions can be used for DoS if the pattern and input requires exponential time to process.\n\nBrakeman will warn about dynamic regular expressions which may be controlled by an attacker. The attacker can create an \"[evil regex](https://www.owasp.org/index.php/Regular_expression_Denial_of_Service_-_ReDoS)\" and then supply input which causes the server to use a large amount of resources.\n\nIt is recommended to avoid interpolating user input into regular expressions.\n"
    },
    "engine_name":"brakeman"
  }
]

Thanks @dzaporozhets.

As alternative to current implementation where json is compared on client side we can do it on server side by downloading artifacts, running diff command and storing result somewhere. Then we send it right to FE.

It will be way faster but more complex since we need to download artifacts on server side, run diff command and save diff result somewhere for further use by Frontend

TODO: try to strip unnecessary info right inside runner with something like

sed -i.bak 's/\,\"content.*/}/' codeclimate.json

Edit: we need to remove only body block so better do something like sed -i.bak 's/\"body\"\:\".*\"//'

cc @ayufan

mentioned in issue gitlab-com/infrastructure#2120

I tried the workaround from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12494. It made the MR page a bit more responsive. But it's still a pain when it's over 10 MB.

Anyway it's a good first step imo.

@JohnnyQQQQ please try this one https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12565. It should be a bit better compared to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12494

@dzaporozhets It went from 25,5 MB raw to 6,2 MB. The browser (Chrome) still gets unresponsive for about 15-20 seconds, but it's much better than before (obviously)

What I also remarked during a look into the codeclimate.json is that there are quit a lot of duplicated entries.

{  
   "check_name":"Controversial/CamelCaseVariableName",
   "fingerprint":"577fda9484f1246369b034b596283862",
   "location":{  
      "path":"in_text.common.php",
      "lines":{  
         "begin":13,
         "end":90
      }
   }
},
{  
   "check_name":"Controversial/CamelCaseVariableName",
   "fingerprint":"577fda9484f1246369b034b596283862",
   "location":{  
      "path":"in_text.common.php",
      "lines":{  
         "begin":13,
         "end":90
      }
   }
} 

I will try to reduce the objects by using the fingerprint. AFAIK it should be unique to the issue reported by the engine.

With

cat raw_codeclimate.json | docker run -i stedolan/jq -c 'map({check_name,fingerprint,location})|unique_by({fingerprint})' > codeclimate.json`

I now get a 1 MB File, which is more than great

Wow. Amazing :)

Another customer experiencing this issue:

ZD: https://gitlab.zendesk.com/agent/tickets/81287

Customer has shared a network profile from the page request. Strangely it appears to load the codeclimate.json file twice in serial, similar to what is described in https://gitlab.com/gitlab-org/gitlab-ee/issues/2910.