Summary
TestDispatchLoop_PartialBatchWaitsForOwnTrigger (internal/ingest/worker_test.go:1092) intermittently fails in t.TempDir() cleanup — after its assertions have already passed — reddening main.
Observed failure
CI run 31423656153 on main @ 3c7d62b:
=== RUN TestDispatchLoop_PartialBatchWaitsForOwnTrigger
=== PAUSE TestDispatchLoop_PartialBatchWaitsForOwnTrigger
=== CONT TestDispatchLoop_PartialBatchWaitsForOwnTrigger
testing.go:1464: TempDir RemoveAll cleanup: unlinkat
/tmp/TestDispatchLoop_PartialBatchWaitsForOwnTrigger2679905009/001/
jetstream/$G/streams/WAVEHOUSE/obs/buffer-consumer: directory not empty
--- FAIL: TestDispatchLoop_PartialBatchWaitsForOwnTrigger (2.06s)
2026/08/10 19:26:33 ERROR double ack failed for processed message
component=ingest_worker error="nats: connection closed" table=events
Re-running the identical commit passed, confirming a race rather than a deterministic break.
Mechanism
The test registers t.TempDir() for the embedded NATS/JetStream data directory. When the test body returns, Go's cleanup runs RemoveAll on that directory — but the ingest worker still has an in-flight goroutine attempting a double-ack (double ack failed ... nats: connection closed). That goroutine is still touching jetstream/$G/streams/WAVEHOUSE/obs/buffer-consumer while RemoveAll walks it, so the unlink races a concurrent write and fails with directory not empty.
Nothing is wrong with the behaviour under test; the assertions pass. Only teardown is unsound.
Why it started showing up now
Likely widened by nats-server 2.14.2 → 2.14.4 in the go-deps group (#438), which can shift shutdown/ack timing. The test itself is unchanged since #192. It passed on #438, #412 and #380 individually and failed on the merged tip, which is consistent with a pre-existing latent race rather than anything introduced by those PRs.
Suggested fix
Make teardown deterministic instead of racing cleanup:
- Have the worker expose a way to wait for its goroutines to drain (e.g. a
WaitGroup the shutdown path joins), and have the test block on that before returning, so t.TempDir cleanup runs against a quiesced server.
- Failing that, don't put the JetStream store under
t.TempDir() — use an explicitly managed directory removed after an confirmed-drained shutdown, so cleanup ordering is under the test's control.
Bumping sleeps would only narrow the window, not close it.
Impact
Low severity, real annoyance: it can fail any main run at random and costs a manual re-run each time. Worth fixing because random red on main trains people to ignore CI.
Summary
TestDispatchLoop_PartialBatchWaitsForOwnTrigger(internal/ingest/worker_test.go:1092) intermittently fails int.TempDir()cleanup — after its assertions have already passed — reddeningmain.Observed failure
CI run 31423656153 on
main@ 3c7d62b:Re-running the identical commit passed, confirming a race rather than a deterministic break.
Mechanism
The test registers
t.TempDir()for the embedded NATS/JetStream data directory. When the test body returns, Go's cleanup runsRemoveAllon that directory — but the ingest worker still has an in-flight goroutine attempting a double-ack (double ack failed ... nats: connection closed). That goroutine is still touchingjetstream/$G/streams/WAVEHOUSE/obs/buffer-consumerwhileRemoveAllwalks it, so the unlink races a concurrent write and fails withdirectory not empty.Nothing is wrong with the behaviour under test; the assertions pass. Only teardown is unsound.
Why it started showing up now
Likely widened by
nats-server2.14.2 → 2.14.4 in the go-deps group (#438), which can shift shutdown/ack timing. The test itself is unchanged since #192. It passed on #438, #412 and #380 individually and failed on the merged tip, which is consistent with a pre-existing latent race rather than anything introduced by those PRs.Suggested fix
Make teardown deterministic instead of racing cleanup:
WaitGroupthe shutdown path joins), and have the test block on that before returning, sot.TempDircleanup runs against a quiesced server.t.TempDir()— use an explicitly managed directory removed after an confirmed-drained shutdown, so cleanup ordering is under the test's control.Bumping sleeps would only narrow the window, not close it.
Impact
Low severity, real annoyance: it can fail any
mainrun at random and costs a manual re-run each time. Worth fixing because random red onmaintrains people to ignore CI.