fix(cluster): elect a primary writer, and let the promotion reach the gate that reads it (#850)

[xe-nvdk]

Summary

IsPrimaryWriter() gates every singleton task: the retention and continuous-query schedulers, and the non-dry-run retention, CQ and delete endpoints. In Pattern 1 (cluster.failover_enabled=true, shared_storage_mode=false, the Enterprise Helm chart's default for local storage) that gate was false on every node, forever, so retention never ran, continuous queries never ran, and those endpoints answered 503 is not primary writer. Nothing logged an error.

Three defects, each sufficient on its own:

• The first promotion could never happen. WriterFailoverManager could only fail over from an existing primary: the health check needed one recorded, HandleWriterUnhealthy returned early unless the node was already primary, and PromoteWriter's only production caller sat inside that path. An initial election now runs when the cluster has no primary, as the compactor manager already did for its lease. It completes without arming the failover cooldown, so a real writer failure in the first minute is not swallowed.
• A promotion never reached the gate. Registry.Get returns a clone, so the callback mutated a copy and re-registered it while IsPrimaryWriter reads c.localNode. That object is now updated on both the promote and demote legs, and on the snapshot-restore path where the FSM carries the state and no callback fires.
• A writer restart lost the designation. A re-join replaces the node's record and the join payload carries no writer state, so the cluster kept a primaryWriterID it could not name; in a single-writer cluster it never recovered, because the only candidate was the node the failover logic had just excluded. The node table now preserves the designation across a re-join, and a healthy writer that is the only candidate can be re-elected.

Also: GET /api/v1/cluster/nodes now reports writer_state (its absence is a large part of why this stayed hidden); a rejected promotion no longer records a designation for a node that does not exist; the manual-failover goroutine is tracked on the WaitGroup Stop joins; the Helm README no longer claims readers can be promoted.

Behaviour change: a write proxied from a reader now goes to the elected primary rather than being spread across healthy writers, which is what Pattern 1 intends. Writers still serve their own traffic directly. Shared-storage clusters are unaffected: failover is suppressed there and the gate keys off Raft leadership.

Out of scope and tracked: #856 (readers are not promotion candidates), #857 (a load balancer cannot target the primary because /ready is role-agnostic), and #858 (a node that leaves gracefully and restarts never re-joins, which I hit while testing this and reproduced identically on origin/main).

Note on #858: with a primary now elected, a node in that state keeps its designation and goes on running singleton work while the cluster no longer lists it. That is a consequence of the membership bug, not of this change, and in Pattern 1 each node owns its storage, which bounds it. It is the strongest argument for fixing #858 next.

Test plan

• Nine tests in internal/cluster/writer_initial_election_test.go: election happens, happens once, is leader-only, does not run without candidates, does not arm the cooldown, survives a re-join, re-elects the sole healthy writer, reaches the local gate on promote and demote, and carries through a restore.
• Revert-run: the four original tests fail on origin/main; the two amended behaviours each fail when that behaviour alone is removed.
• go test -race ./internal/cluster/ ./internal/cluster/raft/, the api cluster tests and a full build: green.
• Live, documented topology (1 writer + 3 readers, failover on): before 503 on every node; after, elected in ~1s, writer_state='primary' in the API, retention execute 200, and a 100s soak with exactly one election and no churn.
• Live restart of the elected primary: it comes back still holding the role and its retention execute returns 200, where origin/main returns 503 both before and after. The same run shows the node failing to re-join the others' view on both builds, which is cluster: a node that leaves gracefully and restarts does not re-join, and the rest of the cluster never sees it again #858.
• Configuration matrix, adversarial plan validator and deep reviewer (in the session); all findings folded in, including the restart deadlock the validator caught.
• CI green

https://claude.ai/code/session_01So2gKWp5TzF9gu3QKNqdeV