WHATWG Streams discussion

[mcollina]

At the Collaborator Summit, I presented a proposal to improve the interoperability of Node.js with WHATWG Streams. You can find my presentation here: https://mcollina.github.io/node-whatwg-streams-proposal/#0.

I would like to gather feedback from this group and move forward with the plan. Maybe we should do a call about this, what do you think? I know this is an area where the Node.js ecosystem could do better.

[littledan]

Thanks for posting this presentation! Is there anything that you can say here about the response at the collaborators summit? Did you get consensus on that statement, "Let's agree that we cannot replace Node.js streams?" Double-buffering definitely seems like an objective technical problem, but it looks like there's a decision point here about whether to support both WHATWG Streams and Node Streams (with the wart of even more incompatible stream types), or whether to use this common pipe/async iterable pattern (with the wart of having some parts of the API not match up). Are the collaborators at any sort of consensus on that decision?

[mcollina]

Thanks for posting this presentation! Is there anything that you can say here about the response at the collaborators summit? Did you get consensus on that statement, "Let's agree that we cannot replace Node.js streams?

Overall yes. As an example, we cannot change the http module without breaking all the web frameworks, and this would be too disruptive for our community. The same goes for fs.createReadStream() and crypto, etc..
Essentially our citgm tool would go all-red with this change.

Double-buffering definitely seems like an objective technical problem, but it looks like there's a decision point here about whether to support both WHATWG Streams and Node Streams (with the wart of even more incompatible stream types), or whether to use this common pipe/async iterable pattern (with the wart of having some parts of the API not match up). Are the collaborators at any sort of consensus on that decision?

There is definitely a decision point coming up in the future. I've proposed three things:

1. ship an official interop module on NPM whatwg-stream similar to readable-stream.
2. add utilities to enable processing of streams using async iterators as the preferred/recommended way, as it's on par with on('data') perf-wise ( readline: add support for async iteration node#23916 (comment)).
3. support a pipeline utility to pipe Node.js streams to WHATWG streams without double buffering.

The approach that I've highlighted is compatible with Node.js LTS strategy: we could support Node.js 6, 8, 10 and 11 if we went down that path.

(For all of you that might not know about readable-stream, it is the official and recommended way to implement a stream in userland across multiple Node.js versions).

[littledan]

Overall yes. As an example, we cannot change the http module without breaking all the web frameworks, and this would be too disruptive for our community.

I think that part is clear. What's less clear is whether it'd be OK to make a second, parallel WHATWG stream implementation, which is output by new APIs like fetch (right?).

[mcollina]

I think that part is clear. What's less clear is whether it'd be OK to make a second, parallel WHATWG stream implementation, which is output by new APIs like fetch (right?).

Even if we shipped that today, it would be at least 2+ years before the usage of that API could be widespread because of our LTS policy. Implementing it in the ecosystem first enable us to iterate quickly and provide support for Node 6, 8 and 10.

As I side note, I'm firmly opposed for fetch to be in core because of the different semantics and expectations between the Browser and Node.js related to caching and security headers. fetch implies a 1-1 between a user and JS VM that Node.js cannot assume. But I do not want to derail this discussion into that direction, so maybe we should open another issue?

[littledan]

(Feel free to open another issue; I don't really have anything intelligent to say about fetch myself.)

[joyeecheung]

ship an official interop module on NPM whatwg-stream similar to readable-stream.

If I understand correctly, this is talking about an npm module implementing the WHATWG streams? (possibly with some additional interop utilities exposed?) Or is it talking about a module that helps with the interop between a WHATWG stream implementation (from somewhere else) and Node.js streams?

[mcollina]

If I understand correctly, this is talking about an npm module implementing the WHATWG streams? (possibly with some additional interop utilities exposed?) Or is it talking about a module that helps with the interop between a WHATWG stream implementation (from somewhere else) and Node.js streams?

I would say both things.

[benjamingr]

From the WHATWG side, @devsnek is working on adding async iterator support to DOM streams and @jakearchibald told me he thinks that could be good interop between Node and whatwg streams.

Slides look great! I agree with the conclusions completely.

Also +1 on adding a utility for buildWrite and better interop for write streams.

[mcollina]

I would like to progress this forward, are there any objections in me asking for a repo in the nodejs org?
Who would like to help?

[joyeecheung]

@mcollina SGTM, do you want to open an issue in the admin repo?

[mcollina]

Yes that's the plan!

[mcollina]

Here it is: nodejs/admin#262