Repository navigation
test(wal): wait for async writes instead of racing the writer goroutine - #566
Merged
Merged
Conversation
TestPurgeInactive_ConcurrentWithWrite failed 19 out of 20 runs on Linux while passing 20 out of 20 on macOS. It is a test bug, not a product bug. Append is asynchronous: it enqueues onto entryChan and returns, while a background goroutine performs the file write and only then increments TotalEntries. The test closed its producer channel once the 100 Appends had been ENQUEUED and immediately asserted TotalEntries != 0, which races the writer goroutine. macOS's scheduler happened to let the writer run; Linux did not. Instrumented the actual values to confirm rather than infer: TotalEntries immediately after <-done: 0 DroppedEntries: 0 TotalEntries after 300ms settle: 100 Nothing was dropped and every entry was written — the test simply looked too early. Replaced the immediate read with waitForEntries, which polls until the writer has recorded the entries or fails with a diagnostic after 5s. The same fix replaces two fixed time.Sleep(50ms) calls in TestPurgeInactive_ActiveFileNotDeleted, which have the identical fragility with a larger margin. Verified in a Linux container: the target test goes from 19/20 failures to 0/20, and the full WAL suite now passes there for the first time. Darwin stays green, including under -race.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #565, where I flagged
TestPurgeInactive_ConcurrentWithWritefailing under Linux containers.It is a test bug, not a product bug — and not flaky
I measured it rather than assuming:
Nearly deterministic on Linux. "Flaky" was the wrong label — the test relies on a race the writer goroutine has no particular reason to win, and macOS's scheduler simply happens to let it.
Root cause
Appendis asynchronous. It enqueues ontoentryChanand returns (wal.go:444-453); a background writer goroutine performs the file write and only then incrementsTotalEntries(wal.go:300).The test closed its producer channel once the 100
Appendcalls had been enqueued, then immediately assertedTotalEntries != 0.<-donesays nothing about whether the writer ran.I instrumented the actual values to confirm rather than infer:
Nothing was dropped and every entry was written. The WAL behaved correctly throughout; the test just looked too early.
Fix
Added
waitForEntries, which polls until the writer has durably recorded the entries, or fails after 5s with a diagnostic (TotalEntries=… dropped=…) so a future failure says why.Also replaced two fixed
time.Sleep(50 * time.Millisecond)calls inTestPurgeInactive_ActiveFileNotDeleted. Those have the identical fragility with a larger margin — a fixed sleep only sets how often the race is lost, and on a loaded CI box or a slow container it can be lost too.Verification
TestPurgeInactive_ConcurrentWithWrite, Linux ×20-race×10Test-only change — one file, no product code touched, so no release-notes entry.
Note the cross-compiled
-racebinary can't be built here (needs a Linux C toolchain), so the container runs are non-race builds. Darwin-racecovers the detector side.