Gzip response bodies

[david-crespo]

Closes #221

Needed a break so I kinda went to town here -- many rounds of review and iteration with both Claude Code and Codex. It's long as hell but it's almost all tests. Leaving as a draft for now because I want to go through it in more detail and write up a better description, but it seems pretty legit.

[david-crespo]

This seems like a way for a dropshot handler to tell dropshot not to compress the response even though it might otherwise compress it. That's neat, but happy to get rid of it if we don't need it.

[david-crespo]

Used for converting between AsyncRead/AsyncWrite from async-compression and the streams used by Body from hyper.

[david-crespo]

This is the primary happy path test for compression.

[ahl]

I gave this a look over and it looks good; let me know if you'd like me to look in more detail. One thing I'm unsure of is how the content-length is set given the streaming compression. It should reflect the compressed size rather than the original size--is that right? Thanks for doing this.

[david-crespo]

I was confused about what is supposed to happen if the client can't handle a streaming response, but it seems all HTTP/1.1 clients must be able to handle chunked transfer encoding:

A recipient MUST be able to parse the chunked transfer coding (Section 4.1) because it plays a crucial role in framing messages when the payload body size is not known in advance.

https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.1

Full answer from GPT-5 through the Codex CLI below. Pretty helpful. The line "HTTP/1.1 clients that advertise Accept-Encoding: gzip are already required to understand chunked responses." is technically true but confusingly phrased because it makes it sound like the requirement is somehow tied to header when it is not.

---

Dropshot only decides to gzip when should_compress_response returns true (dropshot/src/compression.rs:100), which already filters out cases that inherently need fixed-size bodies: no body statuses, HEAD, partial ranges, missing Content-Type, SSE, or content smaller than 512 bytes. Once we do compress, apply_gzip_compression wraps the body in a streaming Gzip encoder and explicitly strips Content-Length/Accept-Ranges (dropshot/src/compression.rs:191 and dropshot/src/compression.rs:238). Hyper will therefore deliver the response with chunked transfer coding on HTTP/1.1, even if the original body was a single buffer. That's the trade-off for not buffering the full compressed entity up front—we never know the final size early enough to set Content-Length.

Why this doesn't break non-streaming clients: HTTP/1.1 clients that advertise Accept-Encoding: gzip are already required to understand chunked responses. Clients that cannot handle streaming should omit Accept-Encoding (they'll stay on the uncompressed path with a fixed length) or explicitly disable compression via the NoCompression extension. Our streaming endpoints continue to stream but remain uncompressed because they typically lack a compressible Content-Type, which should_compress_response uses to short-circuit (dropshot/tests/integration-tests/gzip.rs:344). Tests cover all combinations: buffered responses keep a fixed length (dropshot/tests/integration-tests/streaming.rs:55), streaming endpoints stay chunked (dropshot/tests/integration-tests/streaming.rs:91), and streaming + gzip gets rejected (dropshot/tests/integration-tests/gzip.rs:366).

Matrix (server view vs client capability):

• Server sends fixed-length body → clients that can't stream are fine; clients that can stream are also fine.
• Server sends chunked/streaming body (including gzip) → clients that can stream are fine; clients that can't must not have asked for compression, so they fall back to the first row.

If you have a population of Accept-Encoding: gzip clients that truly cannot process chunked bodies, the short-term mitigation is to strip or downgrade that header in front of Dropshot; longer term you could add an alternate compression path that buffers and sets Content-Length, but that's intentionally avoided in the current design.

[ahl]

Thanks for the clarification. I see in the tests how you're validating the content-length header, etc.

Should compression be configurable in the server settings? i.e. would it make sense to e.g. have compression=on be the default but allow one to make a server that never compresses responses?

[david-crespo]

Honestly not sure. Kind of hard to imagine why you would want to not compress if a client asked for it. But on the other hand it seems heavy-handed to just do it.

[david-crespo]

I added the config option (default true) and I used some helpers to shorten the tests a bit, which to me makes them a little more scannable, though you might prefer things inlined. I think this is ready for a real review.