Skip to content

stream: add collect method

Rodrigo Muino Tomonari requested to merge github/fork/bengl/bengl/stream-collect into master

Returns a promise fulfilling with a sensible representation of the entire stream's data, depending on the stream type.

Buffer streams collect to a single contiguous Buffer. Buffer streams with encoding set collect to a single contiguous string. Object-mode streams collect to an array of objects.

Limiting is allowed, enabling situations like capping request body sizes, or only consuming a limited amount of process.stdin.

Ref: https://github.com/nodejs/node/issues/35192 Ref: https://github.com/nodejs/tooling/issues/68

Rationale

On the web, HTTP response objects have methods like arrayBuffer() and text() to get the full contents of the stream as a single item. In Node.js, tools like the body-parser middleware are often used for server IncomingMessage streams, and tools like concat-stream are used in other places. It would be remarkably handy for users if such a thing were built-in to the Readable prototype.

It's worth mentioning that the stage 2 iterator helpers proposal adds a toArray() method which reads an async iterator to completion in a similar way. That being said, in order to get, for example, a string from a readable stream of buffers, you'd still have to do something like

console.log(Buffer.concat(await readable.toArray()).toString('utf8'));

wheras this PR enables something more like this

readable.setEncoding('utf8');
console.log(await readable.collect());

or

console.log((await readable.collect()).toString());

without waiting for that proposal to be implemented in V8.

This enables one-liners as well. Consider the following one-liner to uppercase stdin.

cat myfile.txt | node -e "(async()=>{const x=[];for await (const c of process.stdin)x.push(c);console.log((''+Buffer.concat(x)).toUpperCase())})()"

With this PR, we can shorten this to the following.

cat myfile.txt | node -e "(async()=>{console.log((''+await process.stdin.collect()).toUpperCase())})()"

Why the name? Rust uses this name for similar functionality and it doesn't appear to clash with the iterator helpers proposal or other methods.

Potential Alternative

It might make sense to add this as a static method on Readable, or a helper function on util, like util.collect() that behaves the way the stream method does in this PR, but also attempting to act appropriately on arbitrary (async) iterables. This would effectively be a slightly more stream-friendly version of the upcoming toArray().

Help?

  • Im not sure the documentation is the best way to describe what's going on, so I'm hoping reviewers can chime in a bit there.
  • Are there edge-cases I'm missing in the test?
  • Most of the work is deferred to the async iteration implementation. Maybe there's a more efficient and/or resilient way to do this?

Merge request reports