Description
Async iterators are a stage 3 proposal for asynchronous pull streams (link) with a v8 implementation arriving very soon.
A new Symbol.asyncIterator is introduced. I believe stream.Readable should support Symbol.asyncIteator which would allow it to be iterated with for...await.
Supporting Symbol.asyncIterator means that readable streams are iterable through the new for await loops. ref.
Meaning we'll be able to do
let body = '';
for await(const data of req) body += data;
const parsed = JSON.parse(body); // note that we can let errors propagate, since we're in an async fn
console.log("got", parsed);Semantics
The async iterator protocol looks like:
interface AsyncIterator {
next(value) : Promise<IteratorResult>
[optional] throw(value) : Promise<IteratorResult>
[optional] return(value) : Promise<IteratorResult>
}
interface IteratorResult {
value : any
done : bool
}Which is similar to the regular JavaScript iteration protocol except that it returns promises. Since it is a pull stream supporting back pressure is pretty simple (since the consumer asks for values rather than being pushed values).
You .next the async iterator to get a promise for the next value in the iteration. You also can build async iterators with async generators. Here are some examples for both from a talk I gave: https://docs.google.com/presentation/d/1r2V1sLG8JSSk8txiLh4wfTkom-BoOsk52FgPBy8o3RM
Implementation
We'll add a Symbol.asyncIterable to readable streams that returns an object with next throw and return methods.
nextis called by thefor awaitloop to get the next value, it expects a promise to be returned that resolves with adone/valuepair. This should likelyreturnis called when a user abruptly completes the loop by callingbreak. This is our chance to do cleanup - like what you'd usefinallyfor and it marks completion (likedone: true).throwis on the consumer's side to propagate errors to the async iterator. I'm not sure what we should do here.
With regards to backpressure - the "easiest" thing would be to pause the stream whenever we just gave a value and unpause it on next - I'm concerend if that might affect performance even further.
Note that while there are interesting problems to solve with async iterators in general - I think our initial focus should be users of the new for await loop.
Performance
Allocating a promise for each value in next sounds prohibitively expensive. However, I am optimistic this will be solved in V8 land and the promise resolution will be inlined.
I recommend we start with a simple implementation and work our way towards a more optimized one.