feat(cluster): report and converge a Raft voter set that disagrees with node roles (#880)

[xe-nvdk]

Closes #880. Follow-up to #862, which made suffrage follow the role at join time and deliberately left existing clusters alone.

Why an upgraded cluster gets stuck

Adding a server that is already a voter as a non-voter updates its address and leaves suffrage untouched — hashicorp/raft's documented behaviour, pinned by an existing Arc test. Most clusters converge anyway via graceful leave → RemoveServer → clean re-add. What does not: a node killed ungracefully, a node whose leave found no leader, the departing leader itself, and any node re-added while still in the configuration.

And there was nothing to look at. A server's suffrage appeared only inside raft.stats.latest_configuration, an opaque fmt.Sprintf dump.

What ships

• GET /api/v1/cluster → raft.membership — per-server suffrage, the role on record, the suffrage that role calls for, whether they agree, plus counts and the node the view came from.
• A rate-limited leader-only warning when mismatches persist, with its own slot.
• POST /api/v1/cluster/voters/converge — admin-only, dry-run by default, demote-only.

The two things worth a reviewer's time

Demote-only is not automatically safe. A configuration change takes effect the moment it is dispatched and commitment is recomputed over the new voter set, so revoking a live voter out of a set whose survivors are dead strands the entry: it can never commit, no further membership change is possible, the leader steps down, and Arc has no RecoverCluster. Every revocation is gated on the liveness of the set it would leave behind, re-checked against what actually remains before each one, and the loop stops at the first failure — deep review caught that the original guard validated only prefixes of the plan, so a mid-plan failure produced a subset nothing had checked. 2→1 voters needs explicit opt-in.

It never promotes: that is the unrecoverable direction, and it is unnecessary because AddVoter on an existing non-voter does promote, so a wrongly-demoted node repairs itself on its next join.

Self is the headline case. Before #862 every joiner was AddVoter'd regardless of role, so a legacy cluster routinely has a reader as Raft leader. Converging the others while the leader is itself a mismatched voter would revoke everyone else's vote, report success, and leave the cluster exactly as it was — a reader leading, IsPrimaryWriter matching nobody, retention/CQ/deletes dead. It now hands leadership to a node whose role votes and returns 409 to re-run.

Live verification — the real upgrade path

A pre-#862 binary was built in a detached worktree, used to form the cluster (where the reader genuinely becomes a voter), then everything SIGKILLed — no leave broadcast — and restarted on this binary over the same Raft data.

legacy:  [{Suffrage:Voter ID:arc-w1} {Suffrage:Voter ID:arc-w2} {Suffrage:Voter ID:arc-r1}]
after:   arc-r1 reader voter matches=false   mismatches: 1
dry run: would_demote [arc-r1]   3 -> 2      (real config untouched)
real:    demoted [arc-r1]        mismatches: 0
         note: "this cluster now has 2 Raft voters; 3 is the documented minimum..."
re-join: arc-r1 SIGKILLed and re-joined -> still a non-voter

And the reader-as-leader case, arranged by bootstrapping a second legacy cluster on the reader: dry run reported would_move_leadership_to: arc-w1 (nothing moved); the real run moved leadership and returned 409; the re-run demoted the reader; and arc-w1 writer_state=primary — that cluster had had no primary writer at all.

The warning fired on a third run left unconverged: 2:06 after upgrade, matching the sustain period, only on the leader.

Test plan

• go build ./..., gofmt -l empty, go vet ./internal/... clean
• go test -race ./internal/cluster/... ./internal/api/... green
• 13 design decisions revert-run individually; all 13 fail without their fix — including the prefix-validation blocker, the dry-run gate, the dedicated warn slot, and admin auth
• Two tests were found passing pre-fix and repaired: voterMismatchCount reads the real Raft config, so a test feeding it a synthetic one asserted nothing; and the partial-failure response had no test
• Configuration matrix written before review, incl. the unauthenticated-status row (no new exposure — the node list and raft.stats already publish roles and suffrage)
• Live: legacy→upgrade→converge, re-join durability, reader-as-leader, sustained warning

Two issues filed and closed during this work — #894 and #892 — both on a false "nothing warns" premise. Arc does warn in both cases; my local rig had ARC_CLUSTER_REPLICATION_ENABLED=false, which disarms WarnIfNoCompactor.

https://claude.ai/code/session_01So2gKWp5TzF9gu3QKNqdeV