[Store] Avoid blocking HA reattachment on master connection retries

[waizuichougou]

Description

Fix the HA client reattachment stall reported in #3740.

Related to #3740.

The failure has two independent causes that must both be addressed:

1. A newly elected master publishes its address before state restoration and
   before the RPC listener is ready. A client can therefore observe the new
   endpoint during the startup window and receive a connection reset. In the
   reported logs, the leader monitor switched to the new address about 0.73
   seconds before the service started accepting connections.
2. After attaching to a master, the HA client continues to use the initial RPC
   connection policy: a 30-second connect timeout with three retries. If the
   deleted Pod IP silently drops SYN packets, one Ping can block for about 120
   seconds. This matches the four reported failover gaps of approximately
   121.001 seconds.

This change fixes the readiness race while preserving leadership fencing:

• The elected master first writes a non-routable
  __mooncake_service_warming__:<lease-id> value into the existing,
  lease-bound master_view key.
• The key remains occupied while the master restores state and starts its RPC
  listener, so another master cannot acquire leadership during warmup while
  clients also cannot mistake the warming value for a ready endpoint.
• The routable address is published only after restoration, service
  registration, leadership renewal, and async_start() have completed.
• Readiness publication uses an etcd transaction that compares the exact
  warming value and preserves the existing lease. This prevents a stale or
  former leader from publishing itself.
• Publication and leadership-loss handling are serialized. If readiness
  publication fails, the RPC server is stopped and the supervisor returns to
  standby.

This change also separates initial connection behavior from HA runtime
behavior:

• Initial HA discovery and configuration loading retain the existing default
  30-second connection timeout and three retries. This preserves resilience
  while the cluster is initially starting.
• After initial attachment, the client enters HA runtime mode before the
  heartbeat and leader monitor start. Runtime connections use zero retries, no
  retry wait, and a one-second default connection timeout.
• MC_RPC_CONNECT_TIMEOUT_MS continues to override the default timeout.
• Reconfiguring the RPC client pool does not invalidate in-flight RPCs.
  Existing calls retain their previous pool instance, while subsequent calls
  use the runtime policy.
• Non-HA clients and store-to-store offload clients keep their existing
  connection behavior.

The change additionally prevents a stale empty master_view watch event from
clearing a newer view installed concurrently by the heartbeat path.

The warming value deliberately uses the same lease-bound election key instead
of a separate unprotected readiness key. This keeps election ownership and
readiness as distinct states while preserving the existing fencing and rolling
upgrade behavior.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Build:

cmake --build build --target rpc_timeout_test high_availability_test -j2

RPC connection-policy tests:

build/mooncake-store/tests/rpc_timeout_test \
  --gtest_color=no --gtest_repeat=3 --gtest_break_on_failure

Results:

• 4/4 tests passed in each of three post-rebase runs.
• The initial HA connection policy retains its existing retry behavior.
• The same client switches to the bounded runtime policy after pool
  reconfiguration.
• Explicit timeout configuration remains covered.

HA readiness and fencing tests:

Run against a real etcd 3.5.21 instance on 127.0.0.1:2379:

GTEST_FILTER='HighAvailabilityTest.*' \
GTEST_COLOR=no \
  build/mooncake-store/tests/high_availability_test \
  --etcd_endpoints=127.0.0.1:2379

Results:

• 14/14 tests passed.
• The tests cover the warming state, publication after service readiness,
  lease preservation, stale-publication rejection, idempotent publication,
  leadership-loss serialization, legacy contender fencing, and the stale
  empty-view race.

Repository checks:

pre-commit run --from-ref upstream/main --to-ref HEAD

• All checks passed locally for the changed files.
• The GitHub Sphinx documentation check passed.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (not applicable; no new user-facing
  option or API is introduced)
• I have added tests to prove my changes are effective
• For changes >500 LOC: not applicable; this change is below the threshold

AI Assistance Disclosure

• No AI tools were used
• AI tools were used.

OpenAI Codex assistance was used for preparing the minimal change and regression test.

[Aionw]

Disabling connection retries cannot fix the underlying problem; it only reduces the stall from multiple attempts to one and may make initial connections less resilient.
I’d prefer separating election ownership from service readiness and publishing the leader endpoint only after the RPC listener is ready.

[waizuichougou]

Thanks @Aionw . I’ll take another look at the code and keep this PR in draft for now. If the reporter shares more details in #3740, I’ll look into them and update the fix accordingly.

[Aionw]

Thanks for your quick reply.