Create an io.js streams team

[rvagg]

@chrisdickinson and I have been charged with splitting streams concerns out of io.js core such that the authoritative source of code, docs and issues for streams comes from this repo and flows back into iojs/io.js. @isaacs has been kind enough to gift this repo to the iojs org and the first order of business is to create a streams team that can be responsible for maintaining this repo, dealing with issues, updating docs, deciding the future directions for streams in io.js (and potentially joyent/node, but we'll see where that goes), and actually implementing it.

Ideally, those involved will have some clue how the current implementation (streams3 mainly) works. The biggest problem iojs (and node) faces wrt streams is the limited number of people that understand how they work, certainly most of the TC doesn't really have an in-depth knowledge.

As a starting point, I'd like to include people that have made contributions to readable-stream since they are the obvious candidates for forming a core team:

• @isaacs
• @rvagg
• @TooTallNate
• @Raynos
• @calvinmetcalf
• @substack
• @sonewman
• @mafintosh
• @timgestson
• @deoxxa
• @maxogden
• @feross

Please indicate your willingness to be included here and then we can start to move forward with some more practical concerns about how this repo & the flow of code, docs & issues is going to work.

[mafintosh]

Great initiative! I'd be more than happy to help out.

[chrisdickinson]

The WHATWG streams spec is shaping up nicely; it might be good to see if @domenic would be interested in helping / peering over what we're working on here depending on his availability.

[calvinmetcalf]

Excited to help (though will not be around much the next week)

On Wed, Dec 31, 2014, 8:23 AM Chris Dickinson notifications@github.com
wrote:

The WHATWG streams spec is shaping up nicely; it might be good to see if
@domenic https://github.com/Domenic would be interested in helping /
peering over what we're working on here depending on his availability.

—
Reply to this email directly or view it on GitHub
#97 (comment).

[sonewman]

I would definitely be happy to be involved 😃

👍 to inviting @domenic, I think that is a good move.

[Fishrock123]

👍 to @domenic

[timgestson]

Would be happy to help.

[rvagg]

I'll schedule a Hangout On Air in a week or two so we can get started, I know that a lot of people have powered down for holidays just at the moment. We can start with the initial list of people here but that doesn't mean it's fixed in stone, we really just need to get some direction and momentum.

[bjouhier]

I am interested if the group is ready to consider some radical departures from current design, more specifically moving from a device-centric approach to a data processing approach. See nodejs/node#188

But otherwise userland is a comfortable place.

[sonewman]

@bjouhier I think there are some interesting ideas and discussions floating around in nodejs/node#89 & nodejs/node#188.

The stream module has had a level 2 unstable rating in joyent/node.

The main issues I see to radical API changes are that a lot of core is built on these constructs, changes to semantics could significantly affect the behaviour of these components and should be approached with caution and not on a whim (how would this implicate movements to merging upstream with joyent/node).

Also despite the unstable rating there is also a large developing eco-system of streams building on top of the current implementation.

IMO the current streams implementations are quite good, although we would benefit from moving closer to ideas from whatwg.

Any changes made would ideally need to be gradually, since they could have significant implications (to core and user-land).

As discussed nodejs/node#89 core's main use-cases for streams are to handle binary and text protocols. User-lands main use-cases are primarily objectMode streams.

For this reason I would advocate making streams as simple, flexible and un-bias as possible.

However this could be a topic for further discussion.

But we should discuss this in another thread.

[Raynos]

You cannot make breaking changes to the streams interface.

WHATWG streams or what @bjouhier proposed (which seems close to simple-streams or pull-streams from creationix & dominictarr) are completely untested in production and will have all kinds of bugs.

Bugs in the streams interface will ripple through everything, including TCP and HTTP. This would devastate the stability of all existing servers in node.

The streams1 to streams2 migration went reasonably smoothly because it had full back compat, but that still introduced a bunch of edge cases.

That being said, count me in for a streams team 👍 !

[bjouhier]

@sonewman The principles that I am proposing also apply to text and binary streams. There are a few text parsers and formatters in ez-streams but we also used the approach to implement a binary TCP/IP protocol converter in our ERP product (and believe me, it was a pretty hairy one).

Separation of concerns applies here: it is easier to implement parsers and formatters with a simple callback API because you don't have to deal with backpressure explicitly.

@Raynos I agree with you about backward compat. Given the complexity of today's streams it seems very risky to reimplement the current interface with a different approach. But streams API could also evolve with a side-by-side approach: keep the current implementation "as is" for backward compat and start fresh on the side.

Saying that simple-streams, pull-streams or ez-streams will have all kinds of bugs is an easy way to rule them out. FWIW we are using ez-streams in our ERP product, which has been released commercially last June. I would not claim that it is bugfree but my experience is that bugs have been very easy to fix so far because the API is very simple.

[mikeal]

Any differences between io.js streams and WHATWG streams will have to be reconciled at some point. By that I do not mean that io.js streams will become WHATWG streams or vice versa, but that we need to iron out any irreconcilable differences while WHATWG streams are being standardized so that either io.js streams become a compatibility API between legacy streams and WHATWG stream or transpilers like browserify can solve this in the future.

The thing I'm most worried about is a situation similar to what we have with Buffer vs TypedArray.

[sonewman]

@mikeal I agree completely, that seems like a perfect direction for discussion and development.

[sonewman]

This would be a good thing to get consensus on: nodejs/node#288