fix(wal): track write failures, attempt rotation, fix nil-file panic

[xe-nvdk]

Fixes #339.

Summary

Three fixes for the WAL writer:

1. Write failures are now tracked. A new FailedWrites counter (exposed in Stats() and as arc_wal_failed_writes_total in Prometheus) lets operators monitor write failure rates. Previously, w.currentFile.Write() errors were logged but not observable in metrics.

2. Rotation is attempted on write failure. If a write fails (disk full, permission change, file deleted), the writer now tries to rotate to a new WAL file and retry the entry. Previously, the same bad file handle kept being used for all subsequent writes.

3. rotate() no longer leaves w.currentFile nil. The rotation function now opens and initializes the new file BEFORE closing the old one. If the new file can't be created (e.g., disk full), the old file handle stays valid — no nil-pointer panic on the next write. This was a pre-existing bug exposed by the new write-error rotation path.

4. Sync errors are now logged. w.currentFile.Sync() errors were silently ignored; they are now logged at ERROR level.

Configuration Matrix

Configuration	Reaches new code?	Preconditions established?
OSS standalone	Yes — WAL writer goroutine runs in all deployments	w.currentFile valid from init rotate(); metrics imported; w.mu held
OSS + compaction	Yes — same	Same
Cluster + no compaction	Yes — same	Same
Cluster + compaction + server.tls_enabled	Yes — same	Same
Cluster + compaction + cluster.tls_enabled	Yes — same	Same

Test Plan

• All 40 WAL tests pass
• Full project build (go build ./...) clean
• gofmt / go vet clean

[gemini-code-assist]

Code Review

This pull request introduces write failure tracking and automatic file rotation to the WAL writer. It adds a FailedWrites counter, exposes it via metrics (including Prometheus), and updates the write path to attempt a file rotation and retry the write upon failure. Additionally, the rotate method is refactored to open the new file and write its header before closing the old file, and sync errors are now captured and logged. The review feedback suggests changing the FailedWrites field in the Writer struct from a raw int64 to atomic.Int64 to prevent potential alignment panics on 32-bit architectures and to align with modern Go idioms, along with corresponding updates to use .Add(1) and .Load().

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[xe-nvdk]

Thanks for the review. Addressing the findings:

FailedWrites → atomic.Int64 (4 comments) — declining, with reasoning

The suggestion is the more modern idiom in the abstract, but applied to this struct it's a net negative:

1. Consistency. Writer has a 6-field block of raw int64 counters — TotalEntries, TotalBytes, TotalSyncs, TotalRotations, DroppedEntries — all accessed via atomic.AddInt64/atomic.LoadInt64. FailedWrites deliberately follows that exact convention. Converting only FailedWrites to atomic.Int64 makes the struct internally inconsistent; converting all six is out of scope for a WAL-bugfix PR.

2. The 32-bit alignment risk doesn't apply here. All six counters are contiguous 8-byte fields in the struct, so they're 64-bit aligned on 32-bit platforms. go test -race ./internal/wal/... passes clean. The alignment footgun is real in general, but it doesn't bite this layout.

So this is a "match the surrounding code" call rather than a correctness fix. If we move to atomic.Int64, it should be a separate sweep that converts the whole counter block at once.

Partial-write-then-rotate (not flagged, but worth noting)

The new retry path (write fails → rotate → re-write the full entry on the new file) can leave a partial entry at the tail of the old file if the first write fails mid-way (e.g. disk fills during the write). I verified this is safe: the WAL reader tolerates a partial trailing entry —

• truncated entry header → io.ReadFull returns ErrUnexpectedEOF, treated as clean EOF (reader.go readEntry);
• truncated payload / checksum mismatch → entry skipped and counted as corrupted, never fatal;
• ReadAll never returns a fatal error for a partial trailing entry, so recovery proceeds with the entries it did read.

No code change needed; I'll add an explanatory comment at the retry site so this isn't re-litigated later.