Add support for sending ReadableStream and WritableStream over RPC, with automatic flow control.#132
Open
Add support for sending ReadableStream and WritableStream over RPC, with automatic flow control.#132
Conversation
🦋 Changeset detectedLatest commit: ef3f6e6 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
Instead of storing an array of RpcStub, we now store an array of the underlying StubHooks. This will make it easier to add support for new types like streams, which aren't RpcStubs, but they will wrap / be wrapped in StubHooks.
(Flow control is left for a future commit.) Written using Claude+Opencode: https://share.opencode.cloudflare.dev/share/gJU0pT8p (I cleaned some stuff up manually.)
As described in protocol.md, the basic idea here is that whenever we want to send a ReadableStream, we first send a message to the other side creating a "pipe". We pump our ReadableStream to the pipe's WritableSteam end, and we deliver the pipe's ReadableStream end to the remote peer. This way, we can begin pushing bytes immediately upon sending a ReadableStream, without waiting for the remote end to call back asking for the bytes (which would be an unnecessary round trip). Written using Claude+Opencode: https://share.opencode.cloudflare.dev/share/ctbbBnOu
If more than 256kb of writes are in flight, we pause writes until past writes complete so that the number drops back below 256kb. Future commits will expand the window size. Written using Claude+Opencode: https://share.opencode.cloudflare.dev/share/D1bqkx2K This was not the best Claude session. I could probably have done it faster by hand.
We add a new "stream" message to the protocol which skips these. See protocol.md for explanation. (Since this is only used for streams, which were just introduced in this PR, this is not a breaking change.) Written using Claude+Opencode: https://share.opencode.cloudflare.dev/share/OfY838e7
kentonv
force-pushed
the
kenton/streams
branch
from
dbad315 to
df03a6a
Compare
With this change, we'll automatically update a stream's window size based on the observed bandwidth-delay product, in order to fully saturate the stream with minimal additional buffer bloat. The algorithm works by observing when each stream chunk is sent and acknowledged (via return from the RPC), allowing us to calculate: 1. Minimum round trip time. 2. Running average bandwidth over the last RTT. From that we calculate bandwidth-delay product and adjust the window to match. We actually set the window a bit bigger than the calculated BDP during startup (2x) and steady-state (1.25x) so that we can observe if the actual bandwidth is greater than expected, and thus update the window accordingly. I worked with Claude+Opencode to design the algorithm and implement, although I significantly refactored almost everything it wrote as the code was pretty meh: https://share.opencode.cloudflare.dev/share/rGV0SKLW
Member
Author
|
Update: I went ahead and added adaptive window size adjustment! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When you send a WritableStream over RPC, the remote side gets a WritableStream. They can write to it. If they write faster than the connection can handle, or faster than your app actually consumes the data, they'll experience backpressure.
When you send a ReadableStream over RPC, the RPC system immediately begins reading from the stream and sending the chunks over the wire so that they are already ready for the remote end to read when it starts reading. This again applies backpressure appropriately. Under the hood, we ask the other end to create a "pipe" -- exposing a WritableStream back to us -- and then we pump chunks into that stream. Meanwhile, the call receiver receives the read end of the pipe.
The flow control is at present based on a fixed window size of 256kb per WritableStream. I intend to fix that in a subsequent change but this is enough to unblock remote bindings that use streams.