HA: a snapshot install that gives up on a database still ACKs the snapshot index, clearing the read floor and enabling stale linearizable reads

[ruispereira]

Summary

When a leader-driven snapshot install "gives up" on a database that has failed to refresh 3 times in a row, the install still ACKs the snapshot index, writes it as applied for every database, clears the stale-read floor and the diverged marks, and returns the installed TermIndex to Ratis (which then purges the log). The node re-enters the ready set advertising fully-caught-up while one database is still stale — so LINEARIZABLE / read-your-writes reads of that database can be served from stale state.

Where (HEAD af4f67fd2)

ha-raft/src/main/java/com/arcadedb/server/ha/raft/DatabaseReconciler.java:

• :99 ACQUIRE_GIVE_UP_AFTER = 3; :313-341 bumpFailureAndShouldRetry returns false once a database has failed 3 consecutive times, so applyReconcileOutcome returns false.
• :270-272 reconcile only fails the install while that returns true:

if (applyReconcileOutcome(plan, outcome))
  throw new IOException("Reconcile from leader had database failure(s); will retry ...");

Past the give-up threshold it returns normally, and ArcadeStateMachine.installSnapshotFromLeader (:1355-1384) unconditionally proceeds:

registerSnapshotMarker(snapshotTerm, snapshotIndex);
lastAppliedIndex.set(snapshotIndex);
writePersistedAppliedIndexForAllDatabases(snapshotIndex);   // :1365 — includes the STALE database
clearStaleSnapshotFloor();                                  // :1369
... raftHA.notifyApplied();
clearDivergedState();                                       // :1380
return installedTermIndex;                                  // Ratis advances applied idx and purges the log

The read gate is global, not per-database: RaftHAServer.getTrustedAppliedIndex (:2834-2838) returns min(getLastAppliedIndex(), floor); with the floor cleared it returns snapshotIndex for reads targeting any database, so waitForAppliedIndex (:2705) / ensureLinearizableFollowerRead (:2967) pass immediately for readIndex <= snapshotIndex.

Failure sequence (3 nodes, follower F behind on database X)

1. Ratis drives InstallSnapshot on F; the refresh of X fails (disk pressure, CRC/size mismatch, transient fault) and install() rolls X back to its old, behind copy.
2. Ratis re-drives; X fails again. On the 3rd consecutive failure reconcile returns normally instead of throwing.
3. The install records snapshotIndex as applied for all databases incl. X, clears the floor and X's diverged mark, and returns to Ratis, which purges the log up to snapshotIndex — the entries needed to advance X are gone from F.
4. F advertises applied = snapshotIndex. A LINEARIZABLE (or RYW with bookmark ≤ snapshotIndex) read of X routed to F passes the wait instantly and is served from the stale copy.
5. Nothing re-arms it: checkStaleFollower sees zero lag, the floor was cleared, checkStuckFollower is term-based. Recovery waits for the next leader change or the next write to X (which trips WALVersionGapException → re-diverge). On an idle X the stale-read window persists.

Suggested fix

Do not treat a given-up per-database failure as a clean install. Any of: (a) keep the affected database in divergedDatabases (clear only databases that actually refreshed) and consult a per-database freshness gate on the read path; (b) refuse to return the installed TermIndex when any planned database did not reach snapshotIndex, so Ratis retries / the node stays NOT_READY; or (c) at minimum keep a per-database "not-at-snapshotIndex" floor so getTrustedAppliedIndex clamps below the missing entries for reads targeting that database (mirroring the #6111 floor, but per-database).

Scope / confidence

Medium-high that the path exists as described (every line verified at HEAD); the stale-read consequence is traced through the read-gate wiring. Not executed — HA is not runnable here. Requires arcadedb.ha.autoAcquireDatabases=true (the default) plus one database whose snapshot install fails ≥3 consecutive times while the leader still reports it readable.

[lvca]

Fixed in #6783.

When a follower syncs from the leader and one database repeatedly fails to install, the node no longer reports itself as fully caught up. That database keeps its own read floor, so a LINEARIZABLE or read-your-writes query against it fails or degrades instead of silently returning stale data, and the node stays out of the ready set until a sync succeeds.

The healthy databases on the same node are unaffected and keep serving normally - that was the point of not retrying the broken one forever. Recovery is retried automatically in the background.

Available in 26.9.1.

❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.