What
POST /v1/ingest now reads the whole request body into a pooled *bytes.Buffer before decoding (internal/api/ingest.go), where it previously streamed: the NDJSON path scanned line-by-line and the array path let json.Decoder compact after each element, so peak resident bytes per request were on the order of one record.
Peak is now O(body) per in-flight request. Two details sharpen it:
bytes.Buffer grows by doubling, so peak allocation can exceed the body cap itself before MaxBytesReader errors.maxPooledBufferBytes (1 MiB) caps what a request may hand back to the pool. It does not cap peak.
There is no semaphore or in-flight-bytes limiter anywhere in internal/api, so the bound is concurrency × maxRequestBodyBytes (16 MiB). Note there is no operator knob for that cap — maxRequestBytes is unexported and test-only — so today the only outer limit is whatever the reverse proxy imposes.
Why it was accepted
Deliberate — it is the shape the native type layer (chtypes) lands against, which needs the body addressable rather than consumed. Documented in the CHANGELOG entry and #540's description rather than silently taken.
Why it still wants a bound
The shipped compose trial policy grants public insert on clicks/events, so a default quickstart deployment is publicly writable. An operator who does not cap body size at their proxy has no backstop, and there is no in-process knob to reach for.
Fix direction
Bound total in-flight body bytes across the ingest handler (a counting semaphore admitting on Content-Length, 503/429 when the budget is exhausted), rather than lowering the per-request cap — the per-request cap is a correctness limit on payload size, not a memory limit.
Alternative if it bites before chtypes lands: reintroduce streaming. The change is localized — newRecordReader is the one place the body shape is chosen, so it means swapping its []byte parameter for an io.Reader and adjusting the two readers behind it.
Raised in review of #540.
What
POST /v1/ingestnow reads the whole request body into a pooled*bytes.Bufferbefore decoding (internal/api/ingest.go), where it previously streamed: the NDJSON path scanned line-by-line and the array path letjson.Decodercompact after each element, so peak resident bytes per request were on the order of one record.Peak is now O(body) per in-flight request. Two details sharpen it:
bytes.Buffergrows by doubling, so peak allocation can exceed the body cap itself beforeMaxBytesReadererrors.maxPooledBufferBytes(1 MiB) caps what a request may hand back to the pool. It does not cap peak.There is no semaphore or in-flight-bytes limiter anywhere in
internal/api, so the bound isconcurrency × maxRequestBodyBytes(16 MiB). Note there is no operator knob for that cap —maxRequestBytesis unexported and test-only — so today the only outer limit is whatever the reverse proxy imposes.Why it was accepted
Deliberate — it is the shape the native type layer (chtypes) lands against, which needs the body addressable rather than consumed. Documented in the CHANGELOG entry and #540's description rather than silently taken.
Why it still wants a bound
The shipped compose trial policy grants
publicinsert onclicks/events, so a default quickstart deployment is publicly writable. An operator who does not cap body size at their proxy has no backstop, and there is no in-process knob to reach for.Fix direction
Bound total in-flight body bytes across the ingest handler (a counting semaphore admitting on
Content-Length,503/429when the budget is exhausted), rather than lowering the per-request cap — the per-request cap is a correctness limit on payload size, not a memory limit.Alternative if it bites before chtypes lands: reintroduce streaming. The change is localized —
newRecordReaderis the one place the body shape is chosen, so it means swapping its[]byteparameter for anio.Readerand adjusting the two readers behind it.Raised in review of #540.