Developer-controlled streams for request/response

[annevk]

First https://github.com/yutakahirano/fetch-with-streams needs to be integrated. So developers can access the stream of request/response. Next is giving them control over the stream.

• What if you do fetch(url, { body }) where body is a stream which gets random stuff (not bytes) enqueued in it?
• Result: when fetch reads from body, it notices a non-BufferSource, and ... does what?
  • Returned promise rejects
  • Cancel the stream
  • On the wire, it just terminates somehow? How does HTTP work?
• Require "cors-with-forced-preflight" (you need a preflight in most cases anyway, and it's also required for progress events, which streams enable) (in particular you need it for everything that isn't a POST with one of the <form> MIME types)
• Always do content-encoding: chunked, and do uploads as chunked encoding, which we need to spec in some way

[mnot]

On the wire, it just terminates somehow? How does HTTP work?

For H1, you'd just close the connection. For H2, you'd cancel the stream.

Always do content-encoding: chunked, and do uploads as chunked encoding, which we need to spec in some way

As noted elsewhere, that's not very interoperable with existing, deployed servers; while some will accept chunked encoding, most will 411 (Length Required) if there's no Content-Length, because they don't want to infinitely buffer the request.

One way around this would be to specify that if Content-Length is set, you'll only write that many bytes to the wire, and error if it tries to write more.

[annevk]

Basic tests: web-platform-tests/wpt#4362. Fetch Standard PR: #425.

[amn]

@annevk: Is chunked a valid value for Content-Encoding HTTP header? I think not? Perhaps you meant Transfer-Encoding: chunked?

[annevk]

@amn yeah, that's what we ended up with in step 4 and 5 of https://fetch.spec.whatwg.org/#concept-http-network-fetch.

[ioquatix]

I have played around with and implemented bi-directional streaming for both HTTP/1 and HTTP/2 in async-http.

While it's not possible to demonstrate in a web browser, here is a simple example of streaming from the server to the client: https://utopia-falcon-heroku.herokuapp.com/beer/index

In addition, using the async-http API directly, it's possible to write applications that stream in both directions, and in theory there is no reason why a web-browser couldn't do it too.

For H1, you'd just close the connection. For H2, you'd cancel the stream.

In some cases you can optimise this logic to avoid closing the connection, but this is roughly what async-http ended up doing.

most will 411 (Length Required) if there's no Content-Length

async-http and by composition, falcon (web server) both accept chunked request body. In addition, falcon accepts non-content-length bodies, but you must use TCP's shutdown(WR) functionality in order to signal the end of the body. With these bits in place, it's possible to have quite decent HTTP/1 bi-directional streaming, even for (a slightly extended at the TCP layer) HTTP/1.0!

I also implemented support for WebSockets, and it's such an orthogonal concept, especially in the design of the server, that I really believe and hope that we can build something better directly on top of HTTP/2. WebSockets seems like a step in the wrong direction.

[ioquatix]

So, what is the status of bi-directional streaming in the browser? Does it work yet?

[annevk]

No.