Skip to content

benchmark: add benchmark for coverage

Rodrigo Muino Tomonari requested to merge github/fork/bcoe/benchmark-coverage into master

Working on #36956, I became a bit worried that as we extend code coverage to support more types of branching, performance might start to become unacceptable.

I created this benchmark to experiment:

Node 14.15.4, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -18.94 % ±0.77% ±1.02% ±1.30%

Node 15.6.0, NODE_V8_COVERAGE, vs., no coverage:

process/coverage.js n=100000 *** -17.95 % ±1.08% ±1.43% ±1.83%

Coverage with #36956, vs., no coverage:

process/coverage.js n=100000 *** -19.37 % ±1.36% ±1.79% ±2.30%

To Summarize

In the current release of 14/15, we see that running Node with coverage decreases performance by about 18%.

With the introduction of optional chaining in this benchmark, we see a 1 - 2% additional hit to performance.

Other interesting facts

  • I benchmarked coverage.js using Istanbul, vs., NODE_V8_COVERAGE, it was 37.46% slower than no coverage.
  • I benchmarked #36956 with no coverage, vs., 15.6.0 with no coverage; there did not appear to be a significant difference in performance (0.37 % ±1.14% ±1.50% ±1.92%).

Conclusions

  • My gut is that it's worth taking the 1 - 2% performance hit to support a new type of branching in our coverage reporting.
  • As new constructs like this are added to JavaScript, we need to be mindful that the cost of branch coverage will continue to increase.
  • I was excited to see that v8's coverage offers a significant performance benefit over pre-instrumenting the JavaScript code.

CC: @schuay

Merge request reports