Consider wrapping Stream.write calls with draining?

[justinwoo]

due to the way Node Streams work, writing to a stream is apparently not even truly fire-and-forget: https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback (e.g. in https://github.com/purescript-node/purescript-node-streams/blob/master/src/Node/Stream.js#L144)

Maybe it might be worthwhile to use their drain event and just return a facade to downstream users for the Eff? e.g. https://github.com/owickstrom/hyper/blob/v0.8.0/src/Hyper/Node/Server.purs#L70

I ran into this while developing my vidtracker app and mashing Ctrl+R...

[hdgarrood]

I'd rather not, as I imagine there are a few tradeoffs to be made with respect to how you actually deal with that. This is supposed to be a low-level library which maps clearly to the Node API. Having said that I'd be interested to see some code if you have a solution in mind; we can probably find a place for it.

[justinwoo]

Oh, and to make things worse, it seems like if you ever to try to write to a buffer anything larger than what it's set to (default 16KB), it will just return false forever. In those cases, I'm finding that creating a buffer of the exact size I need and shoving it into whatever is the only reliable solution...

[hdgarrood]

Do you know how JavaScript developers would address this issue?

[Profpatsch]

What about adding the onDrain callback and maybe providing a convenience wrapper that does both writing and then waits until drain fires?

[hdgarrood]

We should definitely add onDrain, but I think I'd rather avoid convenience wrappers for now.

[rnons]

This is how I'm solving the problem of writing big chunk (way larger than 16KB).

writeBuffer :: forall a. Stream.Writable a -> Buffer -> Effect Unit
writeBuffer stream chunk = do
  -- It's fine if chunk is larger than `highWaterMark`, all data are buffered
  Stream.write stream chunk (pure unit) >>= if _
    then
      Stream.end stream (pure unit)
    else
      -- Trigger the next write on drain, to consume buffer
      onceDrain stream $ Buffer.fromString "" UTF8 >>= writeBuffer stream

While a stream is not draining, calls to write() will buffer chunk, and return false.

https://nodejs.org/dist/latest-v12.x/docs/api/stream.html#stream_writable_write_chunk_encoding_callback

[hdgarrood]

I'm going to close this as we have onDrain now, and I think addressing this is best left to a downstream library.

[jamesdbrock]

Five years later, here's the downstream library which addresses backpressure and draining: https://pursuit.purescript.org/packages/purescript-node-streams-aff/1.1.0/docs/Node.Stream.Aff#v:write