feat(cluster): elect a primary writer even when automatic failover is off (#872)

[xe-nvdk]

Closes #872. Approach chosen by the maintainer (option 1), with the manual hand-over endpoint added at their request.

On local storage, retention, continuous queries and deletes are meant to run on one writer. Choosing which one takes a promotion through Raft, and nothing issued one unless writer failover was both enabled and licensed. With no promotion, IsPrimaryWriter fell back to "any writer is primary" and every writer-role node ran all of it. Three writers meant three nodes executing the same continuous queries and the same deletes.

The licence boundary

A primary is now elected on any local-storage cluster with Raft, whatever cluster.failover_enabled says and whatever the licence contains. What stays paid is what the activation server already names it: "Automatic writer failover", a replacement chosen for you when the primary dies.

The line is enforced on durable state rather than anything a process remembers:

• No designated primary in the state machine → bootstrap → elect, unlicensed.
• A designated primary that is unhealthy → replacement → requires the flag and the licence.

The old code drew this distinction with an in-memory field, which is empty on a freshly started process. A leader restart or a leadership change therefore looked like a cluster that had never had a primary, and it would elect one. That is a replacement under another name, and on an unlicensed cluster it was the paid feature given away for free. A test pins it.

The hand-over endpoint

POST /api/v1/cluster/writers/{id}/demote, admin-only, not gated on the failover licence, because it is how a cluster without automatic failover recovers at all.

It is written as a promotion of somebody else, not a demotion of the node named. A promotion already carries the demotion with it and announces both sides in one applied command; demote-then-elect leaves the election free to pick the same node straight back, which is not a hand-over. With nobody else to promote it releases the designation anyway, so the cluster is not pinned to a writer that may never return, and the response says plainly that nobody took it.

Two blockers the review caught, which this endpoint would have exposed

• A demotion announced nothing. The node being demoted kept believing it was primary and kept running the singleton work, while the cluster still saw a live primary and so never elected anyone. Handing over a healthy writer — the drain-before-upgrade case — was a no-op that reported success.
• Removing the designated primary left the record naming a node that no longer existed. The election only runs when nothing is designated, so it never ran again: no primary, no retention, no continuous queries, no deletes, and nothing reporting why. Removing the dead primary is the obvious operator move, which made this the likely path, and it turns this issue's "work runs N times" into "work never runs" — the harder failure to notice.

Also fixed

A nil-context panic in the failover path, reachable in production because the callback that leads there is wired one line before the manager starts, with health checks already running. A goroutine panic takes the process down.

Behaviour change

On local storage, /ready/write now reports not-ready on a writer that is not the primary. Clusters with automatic failover already behaved this way. Clusters without it reported every writer ready, which was consistent with every writer also running the singleton work, and both were wrong. A hand-rolled load-balancer pool built on that endpoint drops to the primary alone. Kubernetes pod readiness is unaffected, since the chart probes /ready.

One cell this does not fix

cluster.failover_enabled=true with cluster.raft_data_dir empty. There is no Raft to elect through, so the old fallback still applies and every writer is primary. It previously logged nothing at all; it now warns explicitly. Called out because the issue lists three triggers and this fixes two.

Review

Adversarial validation of the plan, then a deep review of the diff.

The validation killed my first boundary: I had claimed a dead primary keeps its designation so an unlicensed cluster would not re-elect. It does not — GetPrimaryWriter filters on health, so the primary dying would have triggered an election with no licence. That is what moved the discriminator to the durable record.

The review then found the two blockers above, plus the panic, a not-leader case matched on message text rather than a sentinel, a configuration that logged nothing, and six places where doc comments and release notes still asserted the behaviour this removes.

Test plan

• go test -race ./internal/cluster/ ./internal/api/ green.
• Revert-run, six behaviours: the licence gate removed; the discriminator back to the in-memory field; the election gated on the licence; removal not releasing the designation; the demotion announcing nothing; and the hand-over as demote-then-elect. Each fails the right test with a message naming what broke.
• Two of those tests initially passed against a restored bug, because a promotion lands asynchronously and they sampled once. They now wait, and the revert-run confirms they fail.
• Live, two writers, failover_enabled unset. Exactly one primary elected. The other writer is refused with "not primary writer"; /ready/write answers 200 and 503 respectively.
• Live, hand-over. From a non-primary: 409 naming the real one. From the primary: 200 reporting the successor. Afterwards both nodes' own views, both readiness probes and the retention gate have all followed the role across.

https://claude.ai/code/session_01So2gKWp5TzF9gu3QKNqdeV