ha-integration-tests red again on main, 5 different Raft ITs, same 'different test each time' pattern as #5668

[robfrank]

Summary

ha-integration-tests was red on main again, with 5 unrelated Raft IT classes each failing with a different symptom in the same run - the same "intermittently red, never the same test twice" pattern as #5668 (closed after #6095/#6112), just recurring.

CI run: https://github.com/ArcadeData/arcadedb/actions/runs/32960831041 (job ha-integration-tests, commit 78e63b0473b74e9aa4c2ff1117eb4ba0d74da343)

Test	Symptom
Issue6221VerifyFanOutGuardIT.aVerifyWithTheClustersRealAddressesStillReportsAgreement	DatabaseComparator$DatabaseAreNotIdentical: Types: DB1 6 <> DB2 5
RaftCommandReadConsistencyIT.readOnlyCommandHonorsReadConsistencyOnFollower	DatabaseIsClosedException: graph
RaftHTTP2ServersCreateReplicatedDatabaseIT.createReplicatedDatabase	IOException: Server returned HTTP response code: 500 for URL: http://127.0.0.1:2480/api/v1/command/graph
RaftPriorityRejoinIT.leaderRestartThenReplicaRestartConverges	ReplicationDispatchedTimeoutException: Group commit entry failed: TimeoutException (entry was dispatched to Raft; outcome unknown)
RaftTimeSeriesWriteReadYourWritesIT.lineProtocolPartialWrite400EmitsUsableCommitIndexBookmark	RemoteException wrapping DatabaseIsClosedException: graph

check-test-results.py reported 5 errors out of 238 tests across 123 failsafe report files; none of the 5 failing classes are adjacent to each other in the run order, and none of the failures repeat a symptom already fixed by #6095/#6112 (those addressed Issue5410AbandonedTicketReleaseIT and RaftReplicationMaterializedViewIT specifically).

Why file a new issue rather than reopening #5668

#5668's suggested direction ("give the HA ITs a shared wait-until-follower-caught-up helper, use it at every cross-node assertion") was implemented for the two call sites it named, but the failures above are new call sites / new symptom shapes:

• Two of the five (RaftCommandReadConsistencyIT, RaftTimeSeriesWriteReadYourWritesIT) throw DatabaseIsClosedException: graph from a live operation, not a stale-read assertion - the database was closed out from under an in-flight command.
• RaftPriorityRejoinIT's ReplicationDispatchedTimeoutException is a genuine dispatch timeout, not an assertion race - it means the underlying operation didn't finish inside its deadline on a loaded runner.
• RaftHTTP2ServersCreateReplicatedDatabaseIT's HTTP 500 and Issue6221VerifyFanOutGuardIT's type-count mismatch are consistent with a follower not yet caught up when checked, which ha-integration-tests is intermittently red on main, on a different test each time #5668's diagnosis covers, but neither of these two classes was in scope for the fix(#5668): deflake RaftReplicationMaterializedViewIT and Issue5410AbandonedTicketReleaseIT #6095/fix(#5668): apply stale-handle fix pattern to RaftReplicationChangeSchemaIT #6112 fix.

Suggested direction

• Audit BaseRaftHATest/BaseGraphServerTest for every cross-node assertion or HTTP call that doesn't already wait for the target follower to catch up to a known commit index/LSN before asserting, and extend the ha-integration-tests is intermittently red on main, on a different test each time #5668 helper to cover them (checkDatabasesAreIdentical/assertClusterConsistency looks like a direct candidate given Issue6221VerifyFanOutGuardIT's failure).
• The two DatabaseIsClosedException: graph failures look like a distinct root cause (a database or server closing earlier than the test expects) rather than a slow-runner race, and are worth investigating separately before assuming they are timing-only.
• Consider whether these 5 tests are individually flaky enough in isolation to warrant a rerun-on-failure policy for this lane, if a root-cause fix per test proves expensive - this session did not attempt to reproduce any of the 5 locally given the cost of standing up multi-node Raft clusters repeatedly.

Related

• ha-integration-tests is intermittently red on main, on a different test each time #5668 (closed) - same "different test each time" pattern, narrower fix

[robfrank]

Fixed in #6776. ha-integration-tests was flaking with DatabaseIsClosedException on two of the reported tests (RaftCommandReadConsistencyIT, RaftTimeSeriesWriteReadYourWritesIT): when an HA follower's database gets closed and reinstalled mid-request during a snapshot resync, the server had no way to tell a client "this is transient, please retry" — it just returned a generic 500. The fix maps that specific condition to an HTTP 503, which the client's existing auto-retry logic already knows how to handle, and updates the flaky test to resolve a fresh database handle on each retry attempt instead of reusing one across reads.

The other three tests reported here (Issue6221VerifyFanOutGuardIT, RaftHTTP2ServersCreateReplicatedDatabaseIT, RaftPriorityRejoinIT) turned out to have different root causes and were not fixed by this PR — they'll need separate investigation, similar to how the earlier #5668 flakiness was worked across multiple PRs.

---

❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.