Skip to content

fix(coordination): source self aux from startup config in SHOW INSTANCE - #4911

Merged
as51340 merged 1 commit into
masterfrom
fix/coord-crash
Sep 25, 2026
Merged

as51340 merged 1 commit into
masterfrom
fix/coord-crash

Conversation

@as51340

@as51340 as51340 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

SHOW INSTANCE on a coordinator that is in the middle of joining the cluster aborted the process with:

Cannot find raft_server::aux for coordinator with id {}.

Root cause

RaftState::GetMyCoordinatorInstanceAux searched the NuRaft cluster config for this coordinator's own entry and asserted when it was missing. That entry is not guaranteed to be there.

CoordinatorStateManager seeds a one-member config containing itself in the constructor, but that is only NuRaft's bootstrap for a fresh server. Both save_config and TryUpdateClusterConfigFromDisk replace cluster_config_ wholesale instead of merging into it, so the seeded self entry lives only until the first real config arrives.

For a joining coordinator the flow is:

  1. Fresh start: seeds a one-member config with itself and runs as its own one-node cluster.
  2. Join request: on ADD COORDINATOR the leader sends a join request carrying its current membership. The follower accepts it and NuRaft calls save_config with that config. It contains the leader and existing coordinators but not the follower, so the self entry disappears in memory and on disk.
  3. Catch-up: the leader streams log entries or a snapshot to the follower until the log gap is small enough.
  4. Membership commit: the leader appends a config that includes the follower and replicates it. Once committed, save_config runs again and self is back.

Between steps 2 and 4 SHOW INSTANCE hit the assertion. A restart inside that window reloads the step 2 config from disk, so the window survives restarts until step 4 completes. Only a follower can hit this, the leader's config always contains itself.

Fix

The state manager keeps its startup network config in my_srv_config_, which is never overwritten. It is exactly what SHOW INSTANCE displays and what AddSelfCoordinator compares the query against, so self aux is now read from there and no longer depends on the cluster config.

  • CoordinatorStateManager::GetMyCoordinatorInstanceAux added, backed by my_srv_config_; aux parsing shared via ParseAux.
  • RaftState::GetMyCoordinatorInstanceAux delegates to it, the assertion is removed.
  • Unit test saves a leader config without this coordinator and checks that self aux still resolves to the startup values.

The cluster config received from the leader may not yet contain the
joining coordinator, so looking up self aux in it could fail and abort
the process. Read it from the state manager's own srv_config instead.
@as51340 as51340 added this to the mg-v3.14.0 milestone Sep 23, 2026
@as51340 as51340 self-assigned this Sep 23, 2026
@as51340 as51340 added CI -build=community -test=core Run community build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push labels Sep 23, 2026
@as51340
as51340 requested a review from andrejtonev September 23, 2026 13:26
@as51340
as51340 marked this pull request as ready for review September 23, 2026 13:26
@as51340
as51340 added this pull request to the merge queue Sep 25, 2026
Merged via the queue into master with commit 1d7a0af Sep 25, 2026
22 checks passed
@as51340
as51340 deleted the fix/coord-crash branch September 25, 2026 11:26
@as51340 as51340 added the Docs - changelog only Docs - changelog only label Oct 2, 2026
@as51340

as51340 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Tracking

  • [Link to Epic/Issue]

Standard development

CI Testing Labels

  • Select the appropriate CI test labels (CI -build=build-name -test=test-suite)

Documentation checklist

  • Add the documentation label
  • Add the bug / feature label
  • Add the milestone for which this feature is intended
    • If not known, set for a later milestone
  • Write a release note, including added/changed clauses
    • SHOW INSTANCE on a coordinator that is in the middle of joining the cluster aborted the process with the error Cannot find raft_server::aux for coordinator with id {}. This is now fixed by returning information about instance's network from the local network config info instead of relying on cluster config which could've been overwritten by the leader's stale information. #4911
  • [] Documentation PR link
    • [] Is back linked to this development PR

@vpavicic vpavicic mentioned this pull request Oct 2, 2026
66 of 75 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI -build=community -test=core Run community build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push Docs - changelog only Docs - changelog only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants