benchmark: add benchmark for coverage
Working on #36956, I became a bit worried that as we extend code coverage to support more types of branching, performance might start to become unacceptable.
I created this benchmark to experiment:
Node 14.15.4, NODE_V8_COVERAGE, vs., no coverage:
process/coverage.js n=100000 *** -18.94 % ±0.77% ±1.02% ±1.30%
Node 15.6.0, NODE_V8_COVERAGE, vs., no coverage:
process/coverage.js n=100000 *** -17.95 % ±1.08% ±1.43% ±1.83%
Coverage with #36956, vs., no coverage:
process/coverage.js n=100000 *** -19.37 % ±1.36% ±1.79% ±2.30%
To Summarize
In the current release of 14/15, we see that running Node with coverage decreases performance by about 18%.
With the introduction of optional chaining in this benchmark, we see a 1 - 2% additional hit to performance.
Other interesting facts
- I benchmarked
coverage.js
using Istanbul, vs., NODE_V8_COVERAGE, it was 37.46% slower than no coverage. - I benchmarked #36956 with no coverage, vs., 15.6.0 with no coverage; there did not appear to be a significant difference in performance (
0.37 % ±1.14% ±1.50% ±1.92%
).
Conclusions
- My gut is that it's worth taking the 1 - 2% performance hit to support a new type of branching in our coverage reporting.
- As new constructs like this are added to JavaScript, we need to be mindful that the cost of branch coverage will continue to increase.
- I was excited to see that v8's coverage offers a significant performance benefit over pre-instrumenting the JavaScript code.
CC: @schuay