Should appliances be pipeable / streams?

[slifty]

Discussion

What do you want to talk about?

This is an item that @chriszs raised months ago and frankly I think I just didn't fully grok at the time, but in light of the recent discussions around payload ordering and backpressure I'm starting to understand more.

The short concept is that right now Appliances (and the countertop in general) use generic events to convey what's going on. The alternative to this is to use streams.

There are many benefits to streams. The most pressing for now:

• They are able to handle backpressure, which means we can control how quickly payloads are read from kafka / generated by source streams.
• They are able to guarantee order across asynchronous pipelines (short term note: it is possible that eventemitters can ensure this too).

As an added bonus we might be able to eliminate the need for custom events.

Relevant Resources / Research

• https://nodejs.org/es/docs/guides/backpressuring-in-streams/
• Other things somewhere.

[slifty]

This is now complete -- Appliances are Transform streams.