Skip to content

fix(cluster): unify delivery topology with Meta CAS + fencing (#5293 #5288) - #5308

Merged
qqeasonchen merged 1 commit into
apache:developfrom
qqeasonchen:develop
Aug 20, 2026
Merged

fix(cluster): unify delivery topology with Meta CAS + fencing (#5293 #5288)#5308
qqeasonchen merged 1 commit into
apache:developfrom
qqeasonchen:develop

Conversation

@qqeasonchen

Copy link
Copy Markdown
Contributor

Motivation

Fixes #5293
Fixes #5288

The uni-architecture cluster delivery currently mixes two modes: sticky delivery with cross-instance HTTP forwarding (HttpForwarder + /internal/forward endpoints) and partition assignment without fencing. The forwarding path adds an extra network hop, hidden failure modes, and makes delivery depend on instance-to-instance reachability. At the same time, partition assignment uses read-then-write generation overwrite, which is racy: a stale instance can silently overwrite the assignment of the rightful owner after a restart or a long GC pause. Finally, ClusterMembership.heartbeat() is never scheduled (#5288), so partition leases expire for live instances.

Modifications

1. Sticky-only delivery topology (issue #5293)

  • Delete HttpForwarder and the /internal/forward + /internal/reply-forward endpoints in UniHttpServer; reply() on a non-owner instance now 404s. The SDK pins to one instance via instanceUrl, so forwarding is no longer needed.
  • Remove DistributionMode.LOAD_BALANCE_STICKY; LOAD_BALANCE absorbs the sticky semantics (partition-key hash routes to one subscriber).

2. Atomic Meta CAS fencing (issue #5293)

  • MetaStore.tryAcquire(key, expectedOldValue, newValue): atomic compare-and-swap. InMemory: ConcurrentHashMap putIfAbsent/replace; Nacos 2.x: publishConfigCas with casMd5 = MD5(expected content).
  • FencingToken "<bootEpoch>:<counter>" per JVM — monotonic and restart-safe. Assignment record /em/assignments/<topic#partition> = "<token>|<ownerInstanceId>".
  • acquireOrFence three-branch protocol:
    • Case 1: unclaimed (or released tombstone "") → CAS claim
    • Case 2: still ours → sync token
    • Case 3: another owner → CAS takeover only when the owner is TTL-evicted or our token is strictly higher; otherwise we are fenced
  • releaseStale: partitions that left our assigner share are CAS'd to a "" tombstone so the new rightful owner can claim them on its next cycle (prevents stranding after membership churn). The tombstone is CAS-equivalent to an absent key on both MetaStore backends.

3. Heartbeat scheduling fix (issue #5288)

  • ClusterMembership.heartbeat() was never scheduled; enableCluster now runs it every 5s and releases it on shutdown together with the partition lease.

Tests

  • FencingTokenTest, InMemoryMetaStoreTest (incl. concurrent CAS), PartitionFencingTest (first claim / race / restart fencing / stale CAS).
  • ClusterDeliveryFaultTest: in-process 3-4 instance fault injection (steady-state split, crash takeover, scale-out churn, Meta partition split-brain guard, healed partition reclaim) driven by a mutable clock — no sleeps, fully deterministic.
  • Removed ClusterForwardIntegrationTest / NacosClusterForwardIntegrationTest (they covered the deleted forwarding path).
  • Full runtime module suite green (189 tests, 0 failures); checkstyle clean.

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (docs/eventmesh-uni-architecture-redesign.md §13.2.10 updated with the acquireOrFence protocol, release path, and fault-injection test coverage)

…#5293 apache#5288)

Sticky-only delivery topology (removes cross-instance forwarding):
- Delete HttpForwarder and the /internal/forward + /internal/reply-forward
  endpoints in UniHttpServer; reply() on a non-owner instance now 404s
  (SDK pins to one instance via instanceUrl, no forwarding needed)
- Remove DistributionMode.LOAD_BALANCE_STICKY; LOAD_BALANCE absorbs the
  sticky semantics (partitionkey hash routes to one subscriber)

Atomic Meta CAS fencing (replaces read-then-write gen overwrite):
- MetaStore.tryAcquire(key, expectedOldValue, newValue): atomic CAS
  (InMemory: ConcurrentHashMap putIfAbsent/replace; Nacos 2.x:
  publishConfigCas with casMd5 = MD5(expected content))
- FencingToken "<bootEpoch>:<counter>" per JVM, monotonic and
  restart-safe; assignment record "/em/assignments/<topic#partition>"
  = "<token>|<ownerInstanceId>"
- acquireOrFence: Case 1 unclaimed (or released tombstone) -> CAS claim;
  Case 2 still ours -> sync token; Case 3 another owner -> CAS takeover
  when TTL-evicted or token strictly higher, else fenced
- releaseStale: partitions that left our assigner share are CAS'd to a
  "" tombstone so the new rightful owner can claim them (prevents
  stranding after membership churn); tombstone is CAS-equivalent to
  absent on both MetaStore backends

Heartbeat scheduling fix (apache#5288):
- ClusterMembership.heartbeat() was never scheduled; enableCluster now
  runs it every 5s and releases it on shutdown together with the
  partition lease

Tests:
- FencingTokenTest, InMemoryMetaStoreTest (incl. concurrent CAS),
  PartitionFencingTest (first claim / race / restart fencing / stale CAS)
- ClusterDeliveryFaultTest: in-process 3-4 instance fault injection
  (steady-state split, crash takeover, scale-out churn, Meta partition
  split-brain guard, healed partition reclaim) driven by a mutable
  clock - no sleeps, fully deterministic
- Remove ClusterForwardIntegrationTest / NacosClusterForwardIntegrationTest
  (they covered the deleted forwarding path)
@qqeasonchen

Copy link
Copy Markdown
Contributor Author

Upstream CI all green on this PR (5 PR-triggered workflows, 9921a4f):

Workflow Result Duration
Continuous Integration (ubuntu + macOS builds)
Code Scanning (java + go) 7m19s + 1m6s
License Check (no new license issues)
dependency-review 2m34s
Greetings 25s

Dependabot Auto-approve shows startup_failure — this is a pre-existing workflow environment issue, not caused by this PR (it fails the same way for any PR to develop).

Same config (License Check + CI + Code Scanning) was previously green on #5307 (which was merged into master as commit b43df620d).

@qqeasonchen
qqeasonchen merged commit 7260581 into apache:develop Aug 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Unify cluster delivery topology and fence stale partition owners [Bug] Start and manage membership heartbeats in sticky cluster mode

1 participant