Skip to content

Optimize child_process IPC for large data

Rodrigo Muino Tomonari requested to merge github/fork/ypresto/optimize-ipc into master

jsonBuffer.indexOf('\n', start) in channel.onread (internal/child_process.js) dramatically slows down when jsonBuffer is very large. jsonBuffer does not contains linebreak before jsonBuffer += decoder.write(pool) call, so checking linebreaks in return value of decoder.write(pool) is enough to parse chunks.

This increases total bytes received in 5 secs by 2-4x in benchmark...!

I found this when developing vscode plugin which returns very large result (around 10MB): https://github.com/rubyide/vscode-ruby/pull/107 Also vscode core has workaround for this bottleneck: https://github.com/Microsoft/vscode/issues/6026#issuecomment-224808371

Also fixed benchmark script which only shows zero.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines
Affected core subsystem(s)

child_process

Benchmark result

Mac OS X 10.12.2 MacBook Pro (Retina, 15-inch, Mid 2014) 2.2 GHz Intel Core i7

Before: child_process/child-process-read-ipc.js dur=5 len=64: 7,583,464 child_process/child-process-read-ipc.js dur=5 len=256: 10,687,553 child_process/child-process-read-ipc.js dur=5 len=1024: 6,279,313 child_process/child-process-read-ipc.js dur=5 len=4096: 6,319,896 child_process/child-process-read-ipc.js dur=5 len=16384: 6,482,388 child_process/child-process-read-ipc.js dur=5 len=65536: 8,580,203 child_process/child-process-read-ipc.js dur=5 len=1048576: 13,398,807 child_process/child-process-read-ipc.js dur=5 len=16777216: 0

After: child_process/child-process-read-ipc.js dur=5 len=64: 9,131,734 child_process/child-process-read-ipc.js dur=5 len=256: 10,502,876 child_process/child-process-read-ipc.js dur=5 len=1024: 6,272,325 child_process/child-process-read-ipc.js dur=5 len=4096: 6,195,482 child_process/child-process-read-ipc.js dur=5 len=16384: 6,505,698 child_process/child-process-read-ipc.js dur=5 len=65536: 8,513,730 child_process/child-process-read-ipc.js dur=5 len=1048576: 40,615,305 child_process/child-process-read-ipc.js dur=5 len=16777216: 30,183,113

When setTimeout() in benchmark is 0: (I have no idea why this produces quite different result.)

Before: child_process/child-process-read-ipc.js dur=5 len=64: 11,924,365 child_process/child-process-read-ipc.js dur=5 len=256: 34,004,413 child_process/child-process-read-ipc.js dur=5 len=1024: 51,082,451 child_process/child-process-read-ipc.js dur=5 len=4096: 61,968,471 child_process/child-process-read-ipc.js dur=5 len=16384: 59,297,771 child_process/child-process-read-ipc.js dur=5 len=65536: 59,533,374 child_process/child-process-read-ipc.js dur=5 len=1048576: 6,704,920 child_process/child-process-read-ipc.js dur=5 len=16777216: 0

After: child_process/child-process-read-ipc.js dur=5 len=64: 12,209,873 child_process/child-process-read-ipc.js dur=5 len=256: 36,964,340 child_process/child-process-read-ipc.js dur=5 len=1024: 50,775,987 child_process/child-process-read-ipc.js dur=5 len=4096: 79,817,420 child_process/child-process-read-ipc.js dur=5 len=16384: 92,115,253 child_process/child-process-read-ipc.js dur=5 len=65536: 105,704,985 child_process/child-process-read-ipc.js dur=5 len=1048576: 23,256,095 child_process/child-process-read-ipc.js dur=5 len=16777216: 0

Merge request reports