Repository navigation
Free Arrow IPC resources from one defer, not two call sites (#733) - #736
Merged
Merged
Conversation
…#733) The Arrow IPC writer released its DuckDB reader, pooled connection and timeout context from two places: straight-line at the end of the writer, and again from safeStream's onPanic. A panic after the straight-line call, in the trailer or logging statements that follow it, ran the whole cleanup a second time. Correcting my own issue report, which claimed this over-releases the reader and logs a spurious cleanup panic. Both are false, measured rather than assumed. duckdb-go's recordReader.Release opens with an explicit refCount <= 0 early return, upstream pins it with a test, *sql.Conn.Close returns ErrConnDone, and cancel is idempotent by contract. Injecting a panic into the writer's final log statement reproduces the double call and produces no cleanup panic and no leak. What was actually wrong is that the code was correct only because all three concrete resources tolerate a second call. The array.RecordReader interface promises nothing of the sort and no test pinned it. Nor would losing the invariant have failed anything: arrow-go guards over-release behind debug.Assert, a no-op unless built with -tags assert, which no Arc build sets, so a second Release would go 0 to -1 and simply not re-run the cleanup branch. Nothing corrupts; nothing reports. An invariant with no signal behind it has to be structural. So cleanup becomes a single defer registered before anything in the writer that can panic. It runs exactly once whether the writer returns normally or unwinds, because a defer inside sw runs during unwinding, ahead of safeStream's recover. This is what every other stream writer in the package already does, so the Arrow IPC path stops being the outlier rather than inventing a shape. It also covers a panic inside onPanic ahead of where the release used to sit, which skipped it and stranded the pooled connection. That one is reasoned, not tested, and says so at the code: nothing in onPanic can be made to panic from a test without adding a seam for it. An earlier draft also claimed runtime.Goexit as a fixed leak. Dropped: it is unreachable on this goroutine outside tests, and forcing it hangs the request regardless, because NewStreamReader never reaches pw.Close. Not strictly better on every path, and the comment says so: releaseArrowStreamResources installs its recover first, so a panic in reader.Release skipped the conn.Close and cancel below it, and a later panic used to retry them through onPanic. That retry is gone. Release now happens after the trailer set and the final log line rather than before them. Those are counter increments, an in-memory map write and one zerolog event, and query_arrow_json.go already holds its connection across the same work. Four stale comments corrected rather than left: releaseArrowStreamResources no longer claims to be called from the panic path, safeStream no longer claims onPanic is for releasing resources, the #716 regression test names the right discriminator, and the note about streamW being the writer's first statement now says "before anything that can panic". The #716 and unreleased, so they now point forward to it. Regression test asserts cleanup runs exactly once on the normal and panic paths, through a new releaseArrowStreamResourcesFunc seam alongside the existing streamArrowIPCFunc one. Verified red on 17aaaef: "cleanup ran 2 times, want exactly 1". Fixes #733
xe-nvdk
force-pushed
the
fix/733-double-release
branch
from
September 11, 2026 23:13
6798461 to
38ddb72
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #733.
What was wrong, and what was not
The Arrow IPC writer released its DuckDB reader, pooled connection and timeout context from two places: straight-line at the end of the writer, and again from
safeStream'sonPanic. A panic after the straight-line call, in the trailer or logging statements that follow it, ran the whole cleanup a second time.I filed #733 claiming this over-releases the reader and logs a spurious cleanup panic. Both are false, and I have corrected the issue. Measured rather than assumed:
duckdb-go'srecordReader.Releaseopens with an explicitrefCount <= 0early return, and upstream pins that with a test.*sql.Conn.Closereturnssql.ErrConnDone, which the helper discards.cancel()is idempotent by contract.Injecting a panic into the writer's final log statement reproduces the double call and produces no cleanup panic, no over-release, and no leak.
Why it is still worth changing
The code was correct only because all three concrete resources tolerate a second call. The
array.RecordReaderinterface promises nothing of the sort and no test pinned it.Nor would losing the invariant have failed anything. arrow-go guards over-release behind
debug.Assert, which is a no-op unless built with-tags assert, and no Arc build sets it, so a secondReleasewould take the refcount from 0 to -1 and simply not re-run the cleanup branch. Nothing corrupts and nothing reports. An invariant with no signal behind it has to be structural rather than conventional.Cleanup is now a single
deferregistered before anything in the writer that can panic. It runs exactly once whether the writer returns normally or unwinds, because a defer insideswruns during unwinding, ahead ofsafeStream's recover.This is what every other stream writer in the package already does (
query.gox3,query_arrow_json.go,query_msgpack.goall defer their release at closure entry and leaveonPanicfor disposition only). Arrow IPC was the outlier; it no longer is.Honest scope of the benefit
It also covers a panic raised inside
onPanicahead of where the release used to sit, which skipped it and stranded the pooled connection. That one is reasoned, not tested, and the code says so: nothing inonPaniccan be made to panic from a test without adding a seam purely for it.An earlier draft of this change also advertised
runtime.Goexitas a fixed leak. I dropped the claim after trying to test it: it is unreachable on that goroutine outsidetesting, and forcing it hangs the request regardless, becauseNewStreamReadernever reachespw.Close(). Claiming an untestable benefit in release notes is the thing this PR's own commit message criticises, so it is gone rather than hand-waved.Not strictly better on every path, and the comment says so.
releaseArrowStreamResourcesinstalls its recover first, so a panic inreader.Releaseskipped theconn.Closeandcancelbelow it, and a later panic used to retry them throughonPanic. That retry is gone. Exotic enough not to be worth restoring, but it is a real direction where the old shape did more.Timing
Release now happens after the trailer set and the final log line rather than before them. Those are counter increments, an in-memory map write and one zerolog event, and
query_arrow_json.goalready holds its connection across the same work.Testing
New
TestExecuteQueryArrowReleasesExactlyOnceasserts exactly one cleanup on the normal and panic paths, through areleaseArrowStreamResourcesFuncseam added alongside the existingstreamArrowIPCFuncone. The panic is injected with a zerolog hook keyed on the final log message, which is the only point landing after the old straight-line release.Verified red on
17aaaef:cleanup ran 2 times, want exactly 1. Green here.go build,go vet,go test -race ./internal/api/all clean, and the CI panic-safety guard passes.Stale docs corrected rather than left
Four, all invalidated by this change and none of which the diff could leave honest:
releaseArrowStreamResourcesno longer claims to be called from the panic path; it has one caller covering both. The note explains why its internal recover must survive anyway.safeStreamno longer saysonPanicis for releasing resources, since no caller does that now. Verified the rewritten wording describes all six current callers.