Data race: Arrow IPC trailers are Set from the stream-writer goroutine while fasthttp serialises the response header

[xe-nvdk]

Found while adding end-to-end coverage for #724. This is pre-existing and affects the trailers shipped by #721/#722, not just the new one.

What races

executeQueryArrow sets its trailers from inside the SetBodyStreamWriter callback:

• internal/api/query_arrow.go — respHeader.Set(arrowExecutionTimeTrailer, ...) on every response
• respHeader.Set(arrowStreamTruncatedTrailer, ...) on the error path
• respHeader.Set(arrowRowsCappedTrailer, ...) on the capped path (A governance row cap is indistinguishable from a complete result #724)

That callback runs on the goroutine fasthttp starts in NewStreamReader (stream.go:42). Meanwhile the connection goroutine serialises the response head: Response.Write → writeBodyStream → ResponseHeader.Write → AppendBytes → appendStatusLine → formatStatusLine.

Both mutate the same ResponseHeader scratch buffer. formatStatusLine writes h.bufKV.value; Set → initHeaderKV writes h.bufKV. Race detector output:

WARNING: DATA RACE
Write at 0x00c0003f0d80 by goroutine 34:
  fasthttp.initHeaderKV()        header.go:3254
  fasthttp.(*ResponseHeader).Set() header.go:1489
  api.(*QueryHandler).executeQueryArrow.func5()  query_arrow.go (trailer Set)
  fasthttp.NewStreamReader.func1()  stream.go:44

Previous write at 0x00c0003f0d80 by goroutine 32:
  fasthttp.formatStatusLine()    status.go:167
  fasthttp.(*ResponseHeader).appendStatusLine() header.go:2366
  fasthttp.(*ResponseHeader).Write()  header.go:2307
  fasthttp.(*Response).writeBodyStream() http.go:1982

Why there is no happens-before edge

NewStreamReader uses fasthttputil.NewPipeConns(), which is buffered: a write from the stream writer does not block waiting for the connection goroutine to read, so it establishes no ordering against the earlier h.Write(w). I confirmed this by making the stream writer write and flush a byte before the Set; the race still fires.

The other direction is fine: Set → bw.Flush() → pw.Close() → reader EOF → writeTrailer is properly ordered, which is why trailer values arrive intact in practice.

Why CI does not catch it today

No test drives executeQueryArrow to its success path over a real HTTP response. TestExecuteQueryArrowReturnsConnectionOnPanic goes through the panic path, and that path deliberately does not set trailers. #722's own comment records the race for the panic path:

By this point fasthttp's connection goroutine may already be serialising the response header, and writing a trailer from this goroutine races with it (caught by the race detector).

The same race exists on the normal path and was never exercised.

Impact

Concurrent writes into a shared slice while the status line is being formatted. Observable as a corrupted status line or corrupted trailer key/value on an Arrow IPC response. Timing-dependent and rare, but not excluded by the memory model.

Why it blocks coverage

Any end-to-end success-path test of POST /api/v1/query/arrow trips it, because Arc-Execution-Time-Ms is set unconditionally. #724 therefore ships with unit coverage of its trailer constant and wiring but no end-to-end assertion that the handler sets it. Fixing this unblocks that test.

Possible directions

There is no race-free way to set a trailer value from the stream-writer goroutine with the fasthttp 1.51 API, since the value has to land in the same ResponseHeader the connection goroutine is serialising. Options worth weighing:

1. Move the signals out of trailers and into the body where the format allows. Not possible for Arrow IPC, which has no envelope.
2. Carry them in pre-body response headers where the value is known before streaming. Works for Arc-Rows-Capped only as "a cap applies", not "the cap was reached".
3. Upstream a synchronised trailer-set API in fasthttp.
4. Serialise access with our own mutex around every ResponseHeader mutation, which only helps if fasthttp's own writes take the same lock. They do not, so this does not actually fix it.

[xe-nvdk]

Two findings from working the fix that change how severe this is, and one dead end worth recording.

The worst case is cross-request corruption, not a torn status line

The issue text caps the impact at "a corrupted status line or corrupted trailer key/value". There is a worse path.

server.go:2412 writes the response, and the connection loop then reaches ctx.Response.Reset() at server.go:2468 with no join on the stream-writer goroutine that NewStreamReader spawned (stream.go:42-49). When the body write fails early, which is an ordinary client disconnect, writeBodyChunked breaks before EOF (http.go:2088) and fasthttp abandons the stream.

The writer goroutine is still alive at that point. It notices only when its own pipe writes start failing, and then runs its tail, which includes the unconditional respHeader.Set(arrowExecutionTimeTrailer, ...). By then the Response has been Reset and, on a keep-alive or pipelined connection, may already belong to the next request. That is a write into another request's response header, not a cosmetic tear on this one.

fasthttp states the rule it was breaking, at http.go:98 in v1.74.0:

Response instance MUST NOT be used from concurrently running goroutines.

Upgrading fasthttp does not fix it

Tested rather than assumed: a scratch bump to fasthttp v1.74.0 (latest) builds clean against Fiber v2.52.13 with -tags=duckdb_arrow, and the race still reproduces, 6 reports, same shape (header.go:1606 Set versus Header()). The scratch buffer was only renamed, h.bufKV becoming h.bufK/h.bufV. No synchronised trailer API was added in 23 releases; Set, AddTrailer and SetTrailer are all plain field writes. So direction 3 in the issue is dead.

Direction 2 is also worse than it looks. Setting a trailer before the body does not give a usable value: header.go:2402-2416 excludes any key registered as a trailer from the header block, so an early Set is still emitted only in the trailer section, carrying the stale pre-body value. It would ship three trailers permanently reading "unknown".

The fix

Publish the trailers from the connection goroutine instead. fasthttp's order there is fixed (http.go:1981-1992): Header.Write, then writeBodyChunked calling bodyStream.Read until io.EOF, then writeTrailer. Since SetBodyStreamWriter(sw) is exactly SetBodyStream(NewStreamReader(sw), -1) (http.go:290), wrapping the stream reader gives a Read that runs on the connection goroutine, after the head is serialised and before the trailers are read back. Ordering by construction rather than by timing.

PR to follow.

[xe-nvdk]

Correcting the issue text: this race has shipped.

The original report said it "arrived with the trailers in this release rather than with the row cap", which put it inside the unreleased 2026.09.2. That is wrong. git log -S 'respHeader.Set(arrowExecutionTimeTrailer' points at 592f1df (2026-05-12), which added Arc-Execution-Time-Ms, and git tag --contains 592f1df gives:

v26.06.1
v26.06.2
v26.06.3
v26.09.1

So the racing Set has been in released builds since v26.06.1, and because the execution-time trailer is set unconditionally, it fires on every Arrow IPC response, not just capped or truncated ones. Only Arc-Stream-Truncated (#722) and Arc-Rows-Capped (#724) are confined to the unreleased cycle.

Consequences for the fix, now applied:

• It gets a proper ## Bug fixes entry in the 2026.09.2 notes naming the affected versions, rather than being treated as a defect introduced and resolved within one unreleased file.
• The upgrade guidance is that nothing needs repairing: both failure modes are transient response corruption, and neither touches stored data.