fix(api): recover from panics in every response stream writer - #720
Merged
Merged
Conversation
fasthttp runs body-stream writers on a bare goroutine (NewStreamReader, stream.go:43): an unrecovered panic there kills the process, and fiber's recover middleware cannot help because the handler has already returned. Only the Arrow IPC writer recovered. The JSON, msgpack, parallel and measurement writers did not, so a panic on any of them stopped the server instead of failing one request. All six shipping writers now go through QueryHandler.safeStream, which recovers, counts the error and logs the panic value, then runs an optional onPanic for work the unwind skipped. onPanic executes inside its own recover: it runs while a panic is in flight, so a second panic there would either kill the process or, being the newest value, replace the root cause in the log. This generalises the shape #716 introduced for the Arrow IPC writer, which now uses the wrapper too. Recovering makes the skipped cleanup matter, so each site's straight-line releases move into a single deferred block preserving the original order. A block rather than separate defers: those run LIFO and would have cancelled the timeout before closing the rows and the pinned profiling connection. Deferring is safe at these sites because none of them writes a response trailer, which is what forced the different treatment in #716. Six sites also disposed of their query-registry entry only on the normal path. Nothing reaps active entries, so a panicking query stayed listed as running forever, holding its SQL and inflating the active-query gauge, and a cancel request would report success against a query that had long finished. Those entries are now failed on the panic path; Fail is a no-op when the normal path already disposed of the entry. A CI step fails the build if a body-stream writer is added without the wrapper. The three writers in arcx_hook.go are excluded: they build only under the arcx_engine tag, which no CI, release or Docker build uses, so they cannot be compile-checked here. Tracked separately. Refs #717
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #717. Deliberately not "Fixes": three writers are out of scope, see the end.
Why this is a crash and not a failed request
fasthttp runs body-stream writers on a bare goroutine:
SetBodyStreamWriter(http.go:290) hands the callback toNewStreamReader(stream.go:29), which doesgo func() { sw(bw); ... }()with no recovery. Fiber's recover middleware is installed but wraps the handler chain, which returned long before the writer runs. So a panic in one of these writers stops the server.Only the Arrow IPC writer recovered (from #716). The JSON, msgpack, parallel-partition and measurement writers did not.
What changed
All six shipping writers now go through
QueryHandler.safeStream, which recovers, counts the error, logs the panic value, and then runs an optionalonPanicfor work the unwind skipped.onPanicruns inside its own recover, because it executes while a panic is already in flight: a second panic there would either kill the process or, being the most recent value, replace the root cause in the only log line describing it. This generalises the shape #716 introduced, andquery_arrow.gonow uses the wrapper rather than its own bespoke recover.Cleanup. Recovering is what makes the skipped cleanup matter, so each site's straight-line releases move into a single deferred block that preserves the original order. A block rather than separate defers: separate ones run LIFO and would have cancelled the timeout context before closing the rows and the pinned profiling connection. Deferring is safe at these sites because none of them writes a response trailer, which is the constraint that forced the different treatment in #716.
A leak the issue did not mention. Six sites disposed of their query-registry entry (
Complete/Fail/TimedOut, or theonComplete/onFailclosures) only on the normal path. Nothing reapsactiveentries, so a panicking query would sit inGET /api/v1/queries/activeasrunningforever, holding its full SQL text, inflating theactive_queriesgauge, and makingDELETE /api/v1/queries/:idreport a successful cancel of a query that finished long ago. Those entries are now failed on the panic path.Failis a no-op when the normal path already disposed of the entry, so the ordering is safe either way.A guard. A CI step fails the build if a body-stream writer is added without the wrapper, so a tenth site cannot quietly reappear.
Test plan
TestSafeStreamIsLoadBearingproves the negative direction in a subprocess: an unwrapped panicking writer kills the test binary, so this cannot be asserted in-process (it takes the whole package down with no per-test attribution).onPanic, the happy path is untouched and does not run it, a panic raised byonPanicitself is contained, and a nilonPanicis fine.TestExecuteQueryReturnsConnectionOnStreamPanicdrives the real handler against a real DuckDB, forces the stream writer to panic, and asserts the pooled connection returns. It carries a vacuity guard that fails the test if the writer never ran, which caught two earlier versions of this test that were proving nothing.InUsestuck at 1.internal/apisuite green under-race; build, vet, gofmt clean.Out of scope, tracked separately
The three writers in
arcx_hook.goare not converted. They build only underarcx_engine, a tag that appears in no CI, release or Docker build, so they ship in no binary and cannot be compile-checked here. Editing them blind would be unverifiable. The CI guard excludes that file with the reason inline. Follow-up issue to come.Silent truncation on the Arrow IPC shape. Recovering converts a truncation the transport could detect into one it cannot: fasthttp still writes the terminating chunk, so the client sees HTTP 200 with a clean body. For JSON that is harmless (the body is unparseable) and for msgpack the length prefixes make it detectable, but an Arrow IPC stream truncated at a record-batch boundary decodes as a short, valid result with no error. That predates this PR on the
query_arrow.gopath and deserves its own fix (a truncation trailer, or closing the connection), not a rushed one here.