Skip to content

fs: improve promise based readFile performance for big files

Rodrigo Muino Tomonari requested to merge github/fork/BridgeAR/improve-fs-promises-readfile into main

This significantly reduces the peak memory for the promise based readFile operation by reusing a single memory chunk after each read and strinigifying that chunk immediately.

Refs: https://github.com/nodejs/node/discussions/44239#discussioncomment-3428693

Signed-off-by: Ruben Bridgewater ruben@bridgewater.de

Benchmark with a few runs:

                                                              confidence improvement accuracy (*)    (**)   (***)
fs/readfile-promises.js concurrent=1 len=1024 duration=5                     -1.18 %       ±7.07%  ±9.67% ±13.12%
fs/readfile-promises.js concurrent=1 len=16777216 duration=5                 -2.11 %       ±3.34%  ±4.56%  ±6.20%
fs/readfile-promises.js concurrent=1 len=33554432 duration=5         ***     59.48 %       ±5.41%  ±7.45% ±10.25%
fs/readfile-promises.js concurrent=1 len=4194304 duration=5          ***      9.95 %       ±5.10%  ±6.93%  ±9.32%
fs/readfile-promises.js concurrent=1 len=524288 duration=5                    1.88 %       ±6.77%  ±9.22% ±12.45%
fs/readfile-promises.js concurrent=1 len=8388608 duration=5           **      6.18 %       ±4.19%  ±5.70%  ±7.69%
fs/readfile-promises.js concurrent=10 len=1024 duration=5                    -1.21 %       ±8.11% ±11.12% ±15.15%
fs/readfile-promises.js concurrent=10 len=16777216 duration=5        ***      9.72 %       ±5.07%  ±6.92%  ±9.36%
fs/readfile-promises.js concurrent=10 len=33554432 duration=5        ***     26.58 %       ±6.29%  ±8.56% ±11.56%
fs/readfile-promises.js concurrent=10 len=4194304 duration=5          **      9.66 %       ±5.88%  ±8.00% ±10.77%
fs/readfile-promises.js concurrent=10 len=524288 duration=5                   1.38 %       ±5.49%  ±7.47% ±10.07%
fs/readfile-promises.js concurrent=10 len=8388608 duration=5         ***     16.84 %       ±4.35%  ±5.93%  ±8.02%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 12 comparisons, you can thus
expect the following amount of false-positive results:
  0.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.12 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)

Merge request reports