feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas (#14183)

* feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas (#13331)

Publisher at the protocol's post-persistence choke point publishes the
mutation's address on the new metadata.mutated cluster channel; peers
converge their ObjectQL registry from their OWN sys_metadata read (write-
through when an active row exists, the delete heal walk when none does),
then replay the event into local onMetadataMutation listeners. The bridge
plugin late-binds attachMetadataMutationPubSub as a second independent
lane at kernel:ready, guarded off the in-process memory driver.

Ruled 2026-09-01 (director batch A, Option A): producer-side fan-out
mirroring the shipped AuthzClusterBridgePlugin shape; the payload is a
signal, never trusted content. Option B (consumer-side self-heal) is not
built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5

* test(metadata-protocol): hold the caller's limit in the fanout find double; ledger the new pinned engine doubles

check:objectql-double-limit named the new find double limit-blind — the
bound now applies after the filter, by presence. The engine-double ledger
learned the fanout file's pinned delete/findOne/update doubles via the
gate's own --write.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5

* docs(permissions): re-anchor the system-context census row moved by the fan-out insertion

check-system-context-census red on CI: the metadata.mutated publisher
block inserted above stripReadonlyForInsert moved the context.isSystem
read 1737 -> 1741, and the census page still anchored the old line. The
gate's own --fix performed the shift (one line-number rewrite, zero
prose); bare re-run green: 109 sites anchored, 145 anchors resolve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5

* docs(permissions): re-derive the census page on the merged tree

The auto-merge of origin/main kept this branch's pre-#14171 anchor rows
beside the row-21 edit, flunking the census 26 ways (paired stale
anchors / unanchored sites in engine.ts and share-link-service.ts —
pure adjacent-row line rot, zero population change). Resolution per the
tool, not by hand: took main's page wholesale, then re-derived with
check-system-context-census --fix, which rewrote exactly ONE anchor
(row 21, protocol.ts 1737 to 1741 — this branch's publisher-block
shift). Delta vs origin/main is that single line; bare gate green: 109
sites anchored, 145 anchors resolve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5

* docs(permissions): re-derive the census page on the second merged tree

Same mechanism as the previous merge: main's #14199 re-anchored 11 rows
of this machine-maintained table while this branch re-anchors row 21,
and a textually clean merge proves nothing about the line numbers.
Resolution per the tool: took main's page wholesale, re-derived with
check-system-context-census --fix on the merged tree — exactly ONE
anchor rewritten (row 21, protocol.ts 1737 to 1741; re-derived, not
carried — main has zero commits on protocol.ts, so the insertion shift
is unchanged). Delta vs origin/main is that single line; bare gate
green (109 sites, 145 anchors). engine-double-contract ledger: zero
main-side commits, gate OK on the merged tree — no regeneration owed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5WBDtaUnoz5XuJ6jk8pQ5

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: claude[bot]

.changeset/metadata-mutation-cluster-fanout.md

@@ -0,0 +1,52 @@
+---
+"@objectstack/metadata-protocol": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas — a runtime-authored object no longer answers OBJECT_NOT_FOUND on every replica that did not perform the write (#13331)
+
+Measured on a live 3-replica EE deployment (ADR-0018 compose, redis driver):
+an object authored through `PUT /api/v1/meta/object/...` persisted to the
+shared `sys_metadata` (so `/api/v1/meta/*` answered 200 fleet-wide) but
+registered with the ObjectQL engine registry of the writing replica only —
+`/api/v1/data/<object>` answered a hard 404 `OBJECT_NOT_FOUND` on the other
+replicas, indefinitely (200 concurrent creates through the LB: 67×201 /
+133×404; a boot-loaded control object: 0 errors; the only recovery was a full
+fleet restart). The runtime authoring path lives entirely in the metadata
+protocol and never touches the metadata service, so the existing
+`metadata.changed` bridge — even when attached — never heard these writes.
+
+Maintainer-ruled design (2026-09-01, Option A):
+
+- **Publisher at the producer choke point.** The protocol's post-persistence
+  mutation funnel (`saveMetaItem` / `publishMetaItem` / `deleteMetaItem` —
+  the same seam `onMetadataMutation` subscribes) now also publishes the
+  mutation's ADDRESS on a new cluster channel `metadata.mutated`
+  (`METADATA_MUTATION_CLUSTER_CHANNEL`, payload
+  `ClusterMetadataMutationPayload`). Drafts are not published — they never
+  enter any replica's registry.
+- **Peers converge from their own DB read.** On receipt, a replica re-reads
+  the row from its OWN `sys_metadata` and re-runs the registry write-through
+  (active row present) or the delete heal walk (no active row). The payload
+  is a signal, never trusted content — the shared database stays the single
+  source of truth, and duplicate or out-of-order delivery converges to the
+  row's current state by construction. After convergence the event replays
+  into the replica's local `onMetadataMutation` listeners (never
+  re-published), so boot-cached consumers such as the authored hook/action
+  re-bind re-sync on peers exactly as they do on the writer.
+- **New attach seam, mirrored from the shipped bridges.**
+  `ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub,
+  nodeId)` — idempotent on the `(pubsub, nodeId)` pair, with loopback
+  suppression via `originNode`, shaped after
+  `MetadataManager.attachClusterPubSub()` and the engine's
+  `attachAuthzInvalidationPubSub()`. `MetadataClusterBridgePlugin` late-binds
+  it at `kernel:ready` as a second, independent lane beside the existing
+  metadata-service lane — the boot shape that lacks a manager-backed
+  `metadata` service (the TS-config host-config boot, exactly the shipped EE
+  shape) is the one that needs this lane most. The new lane skips the
+  in-process memory driver (nothing to fan out to), the guard the authz
+  sibling already carries.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing staleness bound (heal at next boot); this
+channel narrows the window from "until restart" to one network hop.

content/docs/permissions/system-context.mdx

@@ -112,7 +112,7 @@ that silently does not happen.
 | 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10914` |
 | 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:11076` |
 | 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9772` |
-| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1737` |
+| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1741` |
 | 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9809`, `readonly-strict-errors.ts:66` |
 | 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5730` |
 | 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3599`, `:3609`, `:3636` |

packages/metadata-protocol/src/index.ts

@@ -135,6 +135,13 @@ export type { UninstallCleanup, UninstallCleanupOutcome } from './protocol.js';
 // against the producer's contract instead of restating it locally.
 export type { DeletePackageRequest, DeletePackageResponse } from './protocol.js';
 export type { MetadataMutationEvent, MetadataMutationProjector, MutationProjectionOutcome } from './protocol.js';
+// [#13331] The cross-node half of the mutation notification: the cluster
+// channel the protocol publishes post-persistence mutations on, and its
+// address-only payload. Exported for the bridge's tests and for any peer-side
+// consumer that must speak the channel by name — the payload is a SIGNAL,
+// never trusted content (receipt re-reads `sys_metadata` locally).
+export { METADATA_MUTATION_CLUSTER_CHANNEL } from './protocol.js';
+export type { ClusterMetadataMutationPayload } from './protocol.js';
 // [#10219] The per-item publish notification the host bridges to the
 // kernel-wide `metadata:reloaded` announce. Exported for the same reason its
 // mutation sibling is: the subscriber lives in another package.