Skip to content

[Bug] Start and manage membership heartbeats in sticky cluster mode #5288

Description

@qqeasonchen

Execution meta — see plan: #5296

Problem

EventMeshApplication.enableCluster() configures the sticky membership model and creates ClusterMembership, but does not visibly start a periodic membership heartbeat. The periodic call to ClusterMembership.heartbeat() is currently driven by PartitionOwnership.refresh(), while the sticky path intentionally does not wire PartitionOwnership.

As a result, the default sticky-cluster path may not publish/refresh /em/instances/<id> membership records. This affects /session/recommend, load-based instance selection, stale-member expiry, and graceful leave semantics.

Relevant code:

  • eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/boot/EventMeshApplication.java: enableCluster()
  • eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/cluster/ClusterMembership.java: heartbeat()
  • eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/cluster/PartitionOwnership.java: heartbeat scheduling in refresh()

Reproduction

  1. Start EventMeshApplication with clustered Meta configuration.
  2. Call enableCluster() but do not configure PartitionOwnership.
  3. Inspect MetaStore for /em/instances/<instanceId> over multiple TTL intervals.
  4. Query /session/recommend from another instance.
  5. Observe whether membership and load information is published and refreshed.

Expected behavior

Every cluster topology that depends on membership must own an explicit membership lifecycle: immediate startup heartbeat, periodic refresh before TTL expiry, health/degraded behavior when Meta is unavailable, and deletion on graceful shutdown.

Proposed direction

  • Introduce a dedicated membership lifecycle service/scheduler independent of partition ownership.
  • Start it from enableCluster() for sticky mode.
  • Stop it before leave() during shutdown.
  • Define Meta outage behavior for readiness and /session/recommend.

Acceptance criteria

  • Sticky mode writes a membership record immediately after startup.
  • The record timestamp and load snapshot refresh periodically before TTL expiry.
  • Graceful shutdown removes the membership record.
  • Meta write failure is observable and has documented degraded behavior.
  • /session/recommend only uses live membership records.
  • Tests cover sticky mode without PartitionOwnership, TTL expiry, Meta outage, and shutdown.

Suggested labels

bug, reliability, cluster, priority:p0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions