Skip to content

A peer-scoped non-durable message class for plugin traffic, off the replay path #884

Description

@cb1kenobi

Ephemeral plugin traffic needs a way to reach a peer that does not share a backpressure domain with
durable replication. Most of the machinery now exists; what is missing is that it is closed to a
second consumer.

Why not multiplex onto replication

Two reasons, and the second is the real one.

Lifecycle. Connections are keyed per (peer, database), and a plugin channel is not a database.
The generic per-call path opens a fresh short-lived WebSocket per call, which is fine for an
occasional remote operation and unusable per message.

Backpressure. Putting ephemeral traffic on a replication socket puts a publish burst
head-of-line ahead of audit frames and pings on that connection. So a plugin workload could delay,
or past a ping timeout force a reconnect of, durable replication convergence. Ephemeral traffic must
have its own bounded queue whose drops cannot affect replication.

What already exists and should be reused

  • The RPC abstraction, but not the replication socket underneath it. Record locks reach peers
    through registered operations that prefer this worker's live replication subscription session,
    falling back to a per-call connection (replication/recordLockRpc.ts). The registered-operation
    shape is reusable; its preference for the replication session is not.

    Planning review caught this as a contradiction in an earlier draft of this issue, which demanded
    an independent backpressure domain while proposing to generalize a path that rides the
    replication leg. Those are incompatible, for exactly the reason given above: a publish burst on
    that session delays audit frames and pings. Take the abstraction, give ephemeral traffic its own
    connection and its own bounded queue.

  • Capability negotiation. Levels are advertised in the node-name capability bag in
    replication/protocolCapabilities.ts. A plugin relay registers a level there. This retires
    the original version-skew ask, and with it the concern that the inbound dispatch switch has no
    default case.

  • The safety property. Control bytes above 127 stay off the transaction-replay path, which is
    one check. A new non-durable message class provably cannot perturb replay.

What must be built

  • Channel interest and drop accounting on a separate connection. This is the item that matches
    the invariant, and it is the bulk of the work. Planning review pushed back on an earlier draft
    that called opening the record-lock operation union "the cheap part": opening the union is a
    prerequisite for a second consumer, but it is not a smaller version of this option, and
    capability negotiation does not change backpressure. Do not let the cheap item stand in for the
    real one.
  • Open the closed union as well. The record-lock operation type is a closed union of four
    operations and the transport interface core validates is a fixed, lock-specific method set, so a
    second consumer cannot use either today.
  • A channel or topic concept. Subscriptions are database and table filter sets throughout. A
    free-form channel namespace is new, and so is interest propagation. Without interest propagation
    every publish floods every node, and pattern subscriptions are harder still.
  • A unified connected-peers accessor, reduced from the existing node map and connection maps.
  • Bounded queues with drop accounting, so an endpoint can report drops rather than hide them.

Delivery guarantees to state up front

Redis pub/sub is at-most-once and drops for slow or disconnected subscribers, which is a lower
bar than anything else Harper does. Match it deliberately rather than accidentally building
something more durable and slower. Replication pauses under backpressure and replays after
reconnect; both are wrong for ephemeral messages, not merely slow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:replicationReplication, cluster sync, peer connectionsenhancementNew feature or requestfeature:plugin-substratePlugin substrate primitives; Redis endpoint and job queue are the consumers.

    Fields

    Priority

    P3

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions