Skip to content

fix(api): recover from panics in every response stream writer - #720

Merged
xe-nvdk merged 1 commit into
mainfrom
fix/717-stream-writer-panic-recovery
Sep 11, 2026
Merged

xe-nvdk merged 1 commit into
mainfrom
fix/717-stream-writer-panic-recovery

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Sep 11, 2026

Copy link
Copy Markdown
Member

Refs #717. Deliberately not "Fixes": three writers are out of scope, see the end.

Why this is a crash and not a failed request

fasthttp runs body-stream writers on a bare goroutine: SetBodyStreamWriter (http.go:290) hands the callback to NewStreamReader (stream.go:29), which does go func() { sw(bw); ... }() with no recovery. Fiber's recover middleware is installed but wraps the handler chain, which returned long before the writer runs. So a panic in one of these writers stops the server.

Only the Arrow IPC writer recovered (from #716). The JSON, msgpack, parallel-partition and measurement writers did not.

What changed

All six shipping writers now go through QueryHandler.safeStream, which recovers, counts the error, logs the panic value, and then runs an optional onPanic for work the unwind skipped. onPanic runs inside its own recover, because it executes while a panic is already in flight: a second panic there would either kill the process or, being the most recent value, replace the root cause in the only log line describing it. This generalises the shape #716 introduced, and query_arrow.go now uses the wrapper rather than its own bespoke recover.

Cleanup. Recovering is what makes the skipped cleanup matter, so each site's straight-line releases move into a single deferred block that preserves the original order. A block rather than separate defers: separate ones run LIFO and would have cancelled the timeout context before closing the rows and the pinned profiling connection. Deferring is safe at these sites because none of them writes a response trailer, which is the constraint that forced the different treatment in #716.

A leak the issue did not mention. Six sites disposed of their query-registry entry (Complete/Fail/TimedOut, or the onComplete/onFail closures) only on the normal path. Nothing reaps active entries, so a panicking query would sit in GET /api/v1/queries/active as running forever, holding its full SQL text, inflating the active_queries gauge, and making DELETE /api/v1/queries/:id report a successful cancel of a query that finished long ago. Those entries are now failed on the panic path. Fail is a no-op when the normal path already disposed of the entry, so the ordering is safe either way.

A guard. A CI step fails the build if a body-stream writer is added without the wrapper, so a tenth site cannot quietly reappear.

Test plan

  • TestSafeStreamIsLoadBearing proves the negative direction in a subprocess: an unwrapped panicking writer kills the test binary, so this cannot be asserted in-process (it takes the whole package down with no per-test attribution).
  • Wrapper unit tests: recovery runs onPanic, the happy path is untouched and does not run it, a panic raised by onPanic itself is contained, and a nil onPanic is fine.
  • TestExecuteQueryReturnsConnectionOnStreamPanic drives the real handler against a real DuckDB, forces the stream writer to panic, and asserts the pooled connection returns. It carries a vacuity guard that fails the test if the writer never ran, which caught two earlier versions of this test that were proving nothing.
  • Mutation-verified: reverting that site's cleanup to straight-line fails the test with InUse stuck at 1.
  • Full internal/api suite green under -race; build, vet, gofmt clean.
  • Live server across every converted path: JSON, gzip-compressed JSON, msgpack, Arrow IPC and the measurement endpoint all return correct values, and 40 consecutive requests left the pool healthy.

Out of scope, tracked separately

The three writers in arcx_hook.go are not converted. They build only under arcx_engine, a tag that appears in no CI, release or Docker build, so they ship in no binary and cannot be compile-checked here. Editing them blind would be unverifiable. The CI guard excludes that file with the reason inline. Follow-up issue to come.

Silent truncation on the Arrow IPC shape. Recovering converts a truncation the transport could detect into one it cannot: fasthttp still writes the terminating chunk, so the client sees HTTP 200 with a clean body. For JSON that is harmless (the body is unparseable) and for msgpack the length prefixes make it detectable, but an Arrow IPC stream truncated at a record-batch boundary decodes as a short, valid result with no error. That predates this PR on the query_arrow.go path and deserves its own fix (a truncation trailer, or closing the connection), not a rushed one here.

fasthttp runs body-stream writers on a bare goroutine (NewStreamReader,
stream.go:43): an unrecovered panic there kills the process, and fiber's
recover middleware cannot help because the handler has already returned.
Only the Arrow IPC writer recovered. The JSON, msgpack, parallel and
measurement writers did not, so a panic on any of them stopped the
server instead of failing one request.

All six shipping writers now go through QueryHandler.safeStream, which
recovers, counts the error and logs the panic value, then runs an
optional onPanic for work the unwind skipped. onPanic executes inside
its own recover: it runs while a panic is in flight, so a second panic
there would either kill the process or, being the newest value, replace
the root cause in the log. This generalises the shape #716 introduced
for the Arrow IPC writer, which now uses the wrapper too.

Recovering makes the skipped cleanup matter, so each site's straight-line
releases move into a single deferred block preserving the original order.
A block rather than separate defers: those run LIFO and would have
cancelled the timeout before closing the rows and the pinned profiling
connection. Deferring is safe at these sites because none of them writes
a response trailer, which is what forced the different treatment in #716.

Six sites also disposed of their query-registry entry only on the normal
path. Nothing reaps active entries, so a panicking query stayed listed as
running forever, holding its SQL and inflating the active-query gauge,
and a cancel request would report success against a query that had long
finished. Those entries are now failed on the panic path; Fail is a no-op
when the normal path already disposed of the entry.

A CI step fails the build if a body-stream writer is added without the
wrapper. The three writers in arcx_hook.go are excluded: they build only
under the arcx_engine tag, which no CI, release or Docker build uses, so
they cannot be compile-checked here. Tracked separately.

Refs #717
@xe-nvdk
xe-nvdk merged commit a89b33a into main Sep 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant