Repository navigation
runtime: TS-config boot registers a 'metadata' service without attachClusterPubSub — cross-node invalidation disabled; new object gives OBJECT_NOT_FOUND on non-writing replicas, never heals #13331
Description
Activity
Quantified under load on the live 3-replica EE deployment (Traefik LB, redis cluster driver). A concurrency/stress harness driven through the LB (the real production path, not a docker-exec pin) makes the operational severity concrete:
- An object authored at runtime (
PUT /api/v1/meta/object/qa_stress) answers200on/api/v1/meta/object/qa_stresson all 3 replicas (the meta-registry entry broadcasts cluster-wide at create time) — but its/api/v1/data/qa_stressdoor is 200 on the writing replica only, 404 on the other two (the ObjectQL engine registration is replica-local). Directly measured across app-2/3/4:200 / 404 / 404. - Consequently, load through the round-robin LB against a runtime-created object is a ~2/3 hard
404 OBJECT_NOT_FOUND: 200 concurrent creates → 67201/ 133404; authed list reads → ~1/3200. A fleet-wide object (note, boot-loaded on every replica after a restart) under the identical harness is 0 errors, 200/200 creates persisted with 200 distinct rows (no lost writes / dupes), confirming the failure is purely the per-replica registration gap this issue tracks.
So the practical blast radius on a multi-replica deployment is: every object/flow authored at runtime is unusable on
(N-1)/Nof the fleet until a full restart re-registers it from the DB on boot. A restart is the only recovery (boot re-loads all metadata on every replica); there is no runtime propagation of the engine registration.QA-source: #13404 · records-forms/automation (cluster stress) · #13331 attribution
Generated by Claude Code
- An object authored at runtime (
- addedbugSomething isn't workingSomething isn't workingpriority:p0Critical: blocker, must ship before MVPCritical: blocker, must ship before MVP
on Aug 31, 2026 分诊 →
domain:cli· p0 · bug。锚定。 三个候选落点:TS-config 启动路径的插件组装(
packages/runtime/packages/cli)、桥自己穿透 facade(packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts⇒domain:services)、packages/metadata/src/plugin.ts(⇒domain:engine)。根因问题在组装侧(见下),所以主车道domain:cli;跨域半边由该车道 PM 在认领评论里申报完整文件面。定级 p0 —— 依据是 2026-08-31 那条压测评论,不是原文
新读数把它从「一个警告」变成「一条产品线不可用」:
- 通过 LB 的 200 并发创建 → 67 ×
201/ 133 ×404;authed list 读 ~1/3 成功。 - 跨三个副本直接测同一个运行时创建的对象:
/api/v1/data/qa_stress=200 / 404 / 404,而/api/v1/meta/object/qa_stress在三个副本上都是200。 - 对照组成立:boot 时加载的 fleet-wide 对象(
note)在同一套压测下 0 错误,200/200 落库、200 条不同记录。所以失败确实是本卡追的 per-replica 注册缺口,不是压测本身。
判 p0 的四条理由,逐条:① 这是已发货的 EE 多节点路径(ADR-0018 的
objectos-ee/docker-compose.yml)。② 失败是静默且错误方向的 —— metadata API 说对象存在,data API 在 (N-1)/N 的副本上 404,运维看到的是「有时好有时坏」。③ 不自愈:60 秒后重测仍失败,无 TTL、无轮询兜底。④ 没有真绕法 —— 唯一恢复是全队重启,而这恰好取消掉「运行时创作对象/流程」这件功能本身。#13330 有一行 config 绕法所以是 p1;本卡没有,所以是 p0。⛔ 派单前必须先答的问题 —— 卡片把它当成前提,我认为它才是根因
卡片说「TS-config 路径上注册
'metadata'服务的那个东西(runtime dispatcher/domain facade)不暴露attachClusterPubSub」,并把修复设想成给 facade 补一个委托方法。在origin/main上查了一遍,这个前提站不住:ctx.registerService('metadata', …) —— 全树非测试源只有一处: packages/metadata/src/plugin.ts:353⚠️ 这是一次窄查询(字面量 key),对照有命中所以查询本身是好的,但它抓不到变量 key 的循环注册 —— 所以这是部分读数,不是零读数,不能当作结论。但它足以把问题掉个头:MetadataPlugin是全树唯一的'metadata'服务注册方,而卡片实测它根本不在 TS-config 启动的插件阵容里(启动横幅无它,plugin.ts:304的注册日志从不打印)。那么真正要答的不是「facade 为什么不暴露attachClusterPubSub」,而是 —— TS-config 路径本来是不是就该组装MetadataPlugin?它的缺席本身是不是缺陷?如果是,修复是组装侧一行,而且顺带修好所有其他依赖
MetadataPlugin的能力(本卡第二段效应「运行时保存的record_changeflow 只在重启后、且只在本副本上武装」很可能是同一个缺席的另一个面)。如果不是 —— 即 TS-config 路径有意跑另一套 metadata 实现 —— 那才轮到给 facade 补委托方法,而且要问清楚那套实现还少了什么别的。⛔ 不要先写 facade 委托方法。 先跑这个读数:把
serve.ts:2296+只在 artifact 启动时组装MetadataPlugin的那段读掉,答出 TS-config 启动路径上getService('metadata')返回的到底是什么、由谁注册。答案决定修哪个文件,也决定这是domain:cli一条道还是要拉domain:engine半边。排序说明(不是阻塞)
本卡的集群内验证要先有 #13330 的绕法(或修复)才能观察 —— 卡片自己就是这么复现的。但修复本身不被阻塞:落点不同,可以实现并单测。所以不打
pm:blocked;派单时把两张卡交给同一个 dev 连着做最省,验证顺序 #13330 → 本卡。QA 来源:#13404(records-forms/automation 集群压测)已把归因指到本卡。
Generated by Claude Code
- 通过 LB 的 200 并发创建 → 67 ×
CLAIM —
domain:cliexecution PM seat (#6024), sessionsession_01UngCYXF98BVpYA9hfz6NYk.Branch:
claude/issue-13331-metadata-service-cluster-pubsubComments re-read before claiming (3-part protocol): the reporter's stress-test quantification (07:19:25Z) and the triage seat's grading (13:10:05Z). No earlier claim from another session — genuinely unclaimed.
Declared file surface
Per the triage seat's request that the lane PM declare the full surface including cross-domain halves.
⚠️ This surface is provisional by design — the triage seat's ⛔ requires a reading before a fix site is chosen, so the dev may legitimately land in a different column than predicted:path domain expected assembly side of the TS-config boot ( packages/runtime/**)climost likely fix site, if MetadataPlugin's absence is itself the defectpackages/metadata/src/plugin.ts(registerService('metadata', …), ~:353)enginecross-domain half — read first, write only if the facade genuinely runs a second implementation packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts(~:53–59)servicescross-domain half — only if the bridge must reach through the facade packages/cli/src/commands/serve.tscli⛔ EXCLUDED — see below ⛔
serve.tsis excluded from this cardRuling ① — same file is a hard serial with no region exemption.
serve.tsis held by #13330, dispatched concurrently. This card's root-cause reading does point atserve.ts:2296+(the artifact-onlyMetadataPlugincomposition), and reading it is required. Reading is not blocked; writing is. If the reading concludes the repair belongs inserve.ts, the dev must ⛔ stop and report back rather than edit it — I will serialize this card behind #13330 and re-dispatch. Ruling ① is honoured here by assignment, not by hoping the two devs miss each other.The ⛔ that governs the whole dispatch
The card frames "the facade does not expose
attachClusterPubSub" as the premise and a delegate method as the fix. Triage found that premise does not hold up and inverted the question. Carried into the dispatch verbatim:⛔ Do not write the facade delegate method first. First answer: on the TS-config boot path, what does
getService('metadata')actually return, and who registered it? That answer decides which file is edited and whether this is single-lane or needs anenginehalf.⚠️ And the discipline that goes with it: triage'sctx.registerService('metadata', …)sweep returned one non-test site, but it was a literal-key query that cannot see a variable-key or loop registration. Triage labelled it correctly — a partial reading, not a zero. It is a starting point, not a conclusion, and the dev must not treat "only one registrar exists" as established. If the re-measurement needs to stand as a zero anywhere, it needs a positive control that could have returned non-zero.Graded p0 on the 2026-08-31 load measurement, not on the original report: 200 concurrent creates through the LB → 67 ×
201/ 133 ×404, with a fleet-wide object under the identical harness at 0 errors as the control. No workaround exists (#13330 has a one-line one, which is why that card is p1 and this one is p0); the only recovery is a full fleet restart, which cancels the feature itself.
Generated by Claude Code
⚠️ Claim reissued in canonical shape — my earlier comment on this card was MALFORMED and did not register as a claim.It opened
**CLAIM** —(bold, uppercase, em-dash separator). The patrol predicate requires a first line beginning with the literalClaim:; the maintainer's 2026-08-11 ruling makes a differently-written claim malformed, not a dialect, with the repair on the write side and the predicate never widened (check-half-states.mjs,h34ClaimShapedNonCanonicalSeparator). This card — the lane's only p0 — therefore read as unclaimed to the next PM's mutual-exclusion sweep while a dev was on it.Claim: PM loop round 1
Session:session_01UngCYXF98BVpYA9hfz6NYk
Branch:claude/issue-13331-metadata-service-cluster-pubsub
Worktree:objectstack-13331
Domain:domain:cli
File surface:packages/runtime/src/**(expected primary) ·packages/metadata/src/plugin.ts(domain:enginehalf) ·packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts(domain:serviceshalf) · ⛔packages/cli/src/commands/serve.tsREAD-ONLY, excluded from writes (stop on breach; explain in the report)
Container & model:L,mode:subagent,model: opus— tier derived this round bynode scripts/pm/dispatch-gates.mjs --tier packages/runtime/src/index.ts packages/metadata/src/plugin.ts: "no path-derived mandate … floor sonnet · default opus · ceiling fable", ⛔ not recalled
Clause-②: no
Serial constraints cleared:packages/cli/src/commands/serve.tsis held by #13330 — this card reads it (required by triage) but ⛔ may not write it; the dev is instructed to stop and report if the repair lands there, and I will serialize behind #13330 and re-dispatch.⚠️ packages/runtime/src/http-dispatcher.tsis carried by PR #13811, currently in the merge queue — a different file in the same package, so same-package exemption applies and no serial is owed, but the dev should rebase past it. Sibling in-flight: #13330, #13332.Clause-② reasoning: the repair makes cross-node metadata invalidation happen where it was designed to and does not, so it is conformance rather than widening — the same limb the at-tier review of PR #13811 applied ("a door not carrying a declared field was in breach; conformance is not widening").
⚠️ Flips toyesif the fix addsattachClusterPubSubto a facade's published type surface rather than repairing assembly; the dev must measure the built.d.tsboth ways before declaring at PR time, ⛔ not infer it from this line.
Generated by Claude Code
os-dev-report
{ "issue": 13331, "status": "needs_decision", "branch": "claude/issue-13331-metadata-service-cluster-pubsub", "pr": null, "premise_still_valid": false, "summary": "The reading falsifies BOTH the card's premise and the fix its 'no' branch prescribes, so no code was written. (1) WHO REGISTERS THE SLOT: measured on a real ObjectKernel booted in the host-config shape, getService('metadata') returns createMemoryMetadata() — the kernel's in-memory core fallback — registered by ObjectKernel.preInjectCoreFallbacks() at packages/core/src/kernel.ts:270 through a VARIABLE-KEY call this.registerService(serviceName, fallback), which is exactly the shape triage's literal-key query could not see and correctly refused to call a zero. Nothing in packages/runtime registers that slot; MetadataFacade, the obvious candidate, is not installed on main (its own docblock at metadata-facade.ts:134 says so — the contradicting class docblock is filed as #14019). (2) THE YES/NO: NO — MetadataPlugin's absence on the TS-config host-config boot is deliberate, documented three times: isHostConfig ('must NOT be wrapped again'), memory-metadata.ts's own header naming 'the host-config / standalone boot path — where this fallback (not MetadataPlugin) provides the metadata service', and serve.ts:2865 composing MetadataPlugin for host configs in dev ONLY with 'production / cloud host deployments are untouched'. The banner evidence agrees: standalone-stack.ts composes MetadataPlugin unconditionally, so a lineup without it cannot be that stack. (3) WHY THE 'NO' BRANCH ALSO FAILS — the decisive finding: adding attachClusterPubSub to the slot would attach a bridge that is NEVER ASKED. packages/metadata-protocol/src/protocol.ts owns the whole runtime authoring path (saveMetaItem then applyRegistryWriteThrough then applyObjectRegistryMutation then engine.registry.registerObject) and contains ZERO references to cluster, pubsub, or getService('metadata') (measured with two positive controls). metadata.changed's only production publisher is MetadataManager.notifyWatchers, which no PUT /api/v1/meta/* ever reaches on ANY boot shape — ObjectQL's onMetadataMutation listener only does local hook/action rebind and never writes back into the metadata service. So the write persists sys_metadata (shared DB — hence meta reads 200 fleet-wide) and mutates only the LOCAL engine.registry (hence data-plane 200 on the writer, 404 on peers), with no route to the bus. Both faces the card reports are one gap: createMemoryMetadata also has no subscribe, so ObjectQLPlugin (plugin.ts:493) never wires its metadata-to-engine receiver either. The repair is neither in serve.ts nor anywhere in my declared file surface (packages/runtime/**): it needs a publisher at the protocol's onMetadataMutation choke point plus a peer-side registry applier, spanning domain:engine and domain:services, and it mints new public cross-node contract. Escalating rather than widening silently, per the dispatch's own instruction. Independent corroboration: #13609 is open and needs-user-decision for the same seam from a different QA angle ('MetadataManager's unregister DOES broadcast — the observed prolongation sits in an unidentified seam'). This reading identifies that seam.", "tests": "No code changed, so no ablation was run — saying so rather than filling the field with a template. Evidence is measurement, not source-reading alone. (A) TWO-ARM KERNEL EXPERIMENT, direction declared in writing before running. Real ObjectKernel from packages/core/dist, bootstrap(), with a cluster service (driver=redis, nodeId=node-a) and the real MetadataClusterBridgePlugin from packages/services/service-cluster/dist. ARM A (host-config shape, no MetadataPlugin) predicted+observed: _serviceName=metadata, __serviceInfo.status=degraded, typeof attachClusterPubSub=undefined, typeof subscribe=undefined, typeof loadMany=undefined, typeof saveItem=undefined, shape keys = __serviceInfo,_serviceName,register,registerInMemory,get,list,unregister,exists,listNames,getObject,listObjects (byte-identical to the member list in packages/core/src/fallbacks/memory-metadata.ts); attachClusterPubSub calls = 0; bridge emitted the card's line VERBATIM: 'MetadataClusterBridgePlugin: metadata service does not expose attachClusterPubSub(); cross-node cache invalidation disabled'; no attach log. ARM B POSITIVE CONTROL (identical harness, one plugin registers a metadata service that DOES expose attachClusterPubSub): attach called exactly once with {nodeId: node-a, hasPubsub: true}, info log 'bridged metadata.changed to cluster.pubsub (node=node-a)', zero warnings. Verdict line printed: 'READING CONFIRMED (control discriminates)'. This is the distinguisher the bar asked for — Arm A's 'never attached' is a measurement, not a harness artifact, because Arm B attached under the same harness. Incidental confirmation that fallback pre-injection was live in the run: the kernel logged 'CORE: Core service missing, functionality may be degraded: job', the #10746 no-fallback-for-job behaviour. (B) THE ZERO, WITH CONTROLS. Claim: packages/metadata-protocol/src/protocol.ts has no cluster/pubsub/getService('metadata') reference. git grep -cE over that file: no match, exit 1. Control 1 (same instrument, same file, term known present): applyRegistryWriteThrough = 16. Control 2 (same pattern, different file): packages/metadata/src/metadata-manager.ts = 43. Both could have returned non-zero and did, so the zero is a reading. (C) PUBLISHER ENUMERATION: whole-tree sweep of metadata.changed and CLUSTER_CHANNEL shows the only production publish() is metadata-manager.ts:2853 inside notifyWatchers; every other hit is docs, tests, or comments. (D) REGISTRAR SWEEP RE-ESTABLISHED, not inherited: triage's literal-key query reproduced (one non-test site, packages/metadata/src/plugin.ts:353), then widened to registerService( with a NON-literal first argument — which is what surfaced kernel.ts:270 and :303, the registrations the literal query is blind to. (E) BUILDS, both under the shared verify lock: pnpm --filter '@objectstack/service-cluster^...' build (VERDICT command-exit 0, held 126s) and pnpm --filter '@objectstack/service-cluster' build (VERDICT command-exit 0, held 5s). Verdicts read from the lock's own VERDICT line, not a bare exit code. (F) GATE RECONCILIATION: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands, run in the worktree. Provenance line confirmed the answer came from objectstack-ai/objectstack at 62a137baec and that --repo 'holds' against the checkout's origin. Output: 'this branch changes nothing against origin/main (merge base 62a137bae) — nothing to derive'. git status --porcelain empty. named 0, ran 0, unreconciled 0 — a real empty union, because the diff is empty, not because gates were skipped.", "mcp_calls": "4 — one search_issues (dedup, after repo-scoped REST search returned 403 while REST reads returned 200: channel switch declared), two issue_write (the findings below), one add_issue_comment (this report). All card and comment reads went over zero-quota REST.", "open_questions": [ { "question": "How should a runtime-authored metadata item's ObjectQL engine-registry registration reach peer replicas? The existing metadata.changed channel cannot carry it: its only publisher is MetadataManager.notifyWatchers, and the authoring path (protocol.saveMetaItem) never calls the metadata service. This mints new public cross-node contract (a channel, a payload, and an attach entry point) and spans domain:engine plus domain:services, so it is a maintainer/PM call, not a dev one.", "options": [ "A — Fan out the protocol's mutation choke point, mirroring the in-tree authz precedent exactly. The state owner exposes attach-pubsub and a service-cluster bridge late-binds it at kernel:ready, which is literally what AuthzClusterBridgePlugin does today with engine.attachAuthzInvalidationPubSub. Publisher: the protocol's existing onMetadataMutation, already documented as 'the ONE choke point every authoring surface funnels through'. Peer applier: re-run the registry write-through from the peer's OWN sys_metadata read, so peers converge from the source of truth rather than trusting a payload. Touches metadata-protocol/objectql (engine), service-cluster (services), spec (the payload contract).", "B — Self-heal at the consumer: on an assertObjectRegistered miss, fall back to a sys_metadata read before answering 404. No new channel, and it directly addresses the card's 'no TTL/poll fallback on this path'. But it loosens a deliberately fail-closed gate (ADR-0005 / #3770 / #6190) into a tolerant one.", "C — Route runtime authoring through the metadata service so MetadataManager's existing cluster machinery does the work unchanged, which additionally requires giving host-config boots a real manager. Largest blast radius: it reverses the deliberate host-config decision and brings artifact/FS loading semantics into cloud deployments." ], "recommendation": "A, and the four axes do not split on it. REAL BUSINESS NEED: measured, not speculative — 200 concurrent creates through the LB give 67 x 201 / 133 x 404 on a shipped EE path (ADR-0018), with a boot-loaded object at 0 errors as the control; runtime-authored objects ARE the product feature, and today they are unusable on (N-1)/N of any fleet. LONG-TERM SOUNDNESS: A fixes at the producer, so every future authoring transport inherits the signal instead of having to remember it — the tree's own stated rule at protocol.ts:4875, which is also why the publisher must NOT go in the runtime HTTP domain (and therefore why packages/runtime/**, my declared 'most likely fix site', is the wrong column). Reusing the authz bridge's shape means one pattern, not two. Peers converging from their own DB read rather than from the payload keeps the DB the single source of truth. AI-AUTHORED-METADATA SAFETY: A is the only option that keeps the failure loud — B is precisely the consumer-side tolerance that lets an AI-authored object half-exist and the platform paper over it, and it weakens a gate whose fail-closed posture is load-bearing for multi-org correctness, not just for tidiness. STARTUP SCOPE: A repairs a shipped p0 on an already-published path; it adds no speculative surface, and its new contract is one channel plus one attach method modelled on one that already ships. B is cheaper but treats the symptom and taxes a security-relevant gate forever; C is the largest change and collides with #10746's ruling that a fallback must not fake capability, since it would make a real dependency out of a slot the kernel sometimes fills with a degraded fake. If a stopgap is wanted for the current EE fleet while A is built, B is defensible as an explicitly temporary, loudly-logged complement to A — but not as the fix.", "also_needs_a_ruling": "Scope split. A spans domain:engine and domain:services and touches no domain:cli file, so this card as labelled (domain:cli, file surface packages/runtime/**) cannot carry it. Recommend re-labelling and re-dispatching to the engine lane, and folding in #13609 — it is open, needs-user-decision, and is the same seam observed from the delete side ('MetadataManager's unregister DOES broadcast — the observed prolongation sits in an unidentified seam'). One fix closes both." } ], "out_of_scope_findings": [ "filed as #14019: MetadataFacade's class docblock claims it is 'Registered as the metadata kernel service' while the same file at :134 says nothing on main installs it — the likely source of this card's inverted premise", "filed as #14021: MetadataClusterBridgePlugin logs 'bridged' on the in-process memory driver that fans out to nobody (its sibling AuthzClusterBridgePlugin guards exactly this with isInProcessClusterDriver), and has no test file at all" ], "notes": "serve.ts was READ, never written — the exclusion held. My reading does not conclude the repair belongs there: serve.ts:2851-2899 is dev-only HMR scaffolding and is not on the EE production path. Worktree removed cleanly (git status --porcelain empty, no --force). The branch was pushed empty as the write-route probe and carries no commits." }
Generated by Claude Code
PM review — ESCALATING.
domain:cli→domain:engine,pm:dispatched→needs-user-decision. ⛔ No code was written, and that was correct.domain:cliexecution PM seat (#6024) · sessionsession_01UngCYXF98BVpYA9hfz6NYkThe dispatch's ⛔ was "do not write the facade delegate method first — first produce a reading." The reading came back and falsified both branches of the question, so nothing was written. That is the dispatch working, not failing.
⭐ The premise is false, and triage's caution is what caught it
The card says the TS-config path's
'metadata'service is a runtime dispatcher/domain facade lackingattachClusterPubSub. Measured on a real bootedObjectKernel, it is not:getService('metadata')returnscreateMemoryMetadata()— the kernel's in-memory core fallback — registered byObjectKernel.preInjectCoreFallbacks()atpackages/core/src/kernel.ts:270through a variable-key callthis.registerService(serviceName, fallback).⭐ That is exactly the shape the triage seat said its literal-key query could not see, which is why it labelled its own result 「部分读数,不是零读数」 rather than a census. Had that caution not been written down, the obvious next step was to conclude "only one registrar exists" and build the facade delegate on top of a false premise. The refusal to call a partial reading a zero is what prevented a wrong fix.
The yes/no question: NO, and it is documented three times
MetadataPlugin's absence on the host-config boot is deliberate, not an oversight —isHostConfiginplugin-detection.ts("must NOT be wrapped again"),packages/core/src/fallbacks/memory-metadata.ts's own header naming the host-config path as the one this fallback serves instead ofMetadataPlugin, andserve.ts:2865composing it for host configs in dev only ("production / cloud host deployments are untouched").⭐ The decisive finding — the card's own fallback fix would have been a no-op
This is the part that would not have been found by reasoning from the card:
Adding
attachClusterPubSubto the slot would attach a bridge that is never asked.packages/metadata-protocol/src/protocol.tsowns the entire runtime authoring path (saveMetaItem→applyRegistryWriteThrough→applyObjectRegistryMutation→engine.registry.registerObject) and contains zero references to cluster, pubsub, orgetService('metadata').metadata.changed's only production publisher isMetadataManager.notifyWatchers(metadata-manager.ts:2853), which noPUT /api/v1/meta/*reaches on any boot shape.⇒ The write persists
sys_metadata(shared DB — hence meta reads 200 fleet-wide) and mutates only the localengine.registry(hence data-plane 200 on the writer, 404 on peers), with no route to the bus at all. Both faces the card reports are one gap. ⇒ A seat that had "fixed" the facade would have shipped a change, seen the warning disappear, and left the 404s exactly where they were.The evidence bar was met, not asserted
- Two-arm kernel experiment, direction declared in writing first. Arm A (host-config shape):
attachClusterPubSubcalls = 0, and the bridge emitted the card's warning verbatim. Arm B, positive control, identical harness with a metadata service that does expose it: attach called exactly once, info log emitted, zero warnings. ⇒ Arm A's "never attached" is a measurement, not a harness artefact — the control discriminates. - The zero carries two controls.
protocol.tscluster/pubsub sweep returns nothing; control 1 (same instrument, same file,applyRegistryWriteThrough) = 16; control 2 (same pattern,metadata-manager.ts) = 43. Both could have returned non-zero and did. - The registrar sweep was re-established, not inherited — triage's literal query reproduced first, then widened to non-literal first arguments, which is what surfaced
kernel.ts:270.
⛔ The
serve.tsexclusion held, and the conclusion vindicates itserve.tswas read, never written. And the reading concluded the repair does not belong there:serve.ts:2851-2899is dev-only HMR scaffolding, not on the EE production path. ⇒ The serial I imposed cost nothing, and #13330 keeps sole ownership of that file.
Why this becomes
needs-user-decisionrather than a re-dispatchThe repair needs a publisher at the protocol's
onMetadataMutationchoke point plus a peer-side registry applier. That mints new public cross-node contract — a channel, a payload, and an attach entry point — and spansdomain:engine+domain:services, touching nodomain:clifile. Two consequences:- Clause ② fires on content. New public cross-node contract is a widening, and clause ② bars enqueueing. ⛔ So I cannot put this back in a queue, and I am not going to route it through my lane as though the label were the question.
- This card as labelled cannot carry it. Re-labelled
domain:engine, unassigned, so the engine lane can claim it cleanly.
⚠️ It stayspriority:p0. The grading rests on the 2026-08-31 load measurement (200 concurrent creates → 67 ×201/ 133 ×404, with a boot-loaded object at 0 errors as control), and nothing in this reading weakens it — the reading makes the defect deeper, not smaller.四棱分析 —— 四面不分歧,一致指向 A
A 在 protocol 的
onMetadataMutation扇出,照搬树内AuthzClusterBridgePlugin的既有形状(state owner 暴露 attach-pubsub,service-cluster 在kernel:ready晚绑),peer 从自己的sys_metadata读回重放。B 消费侧自愈:assertObjectRegistered落空时回落读一次 DB。C 让运行时创作走 metadata service。- 实际业务需求:已实测,不是设想 —— 已发货的 EE 路径(ADR-0018)上 (N-1)/N 副本不可用,而「运行时创作对象」正是产品功能本身。
- 项目长远合理性:A 修在生产者侧,所以未来每一个创作传输面自动继承这个信号,而不必各自记得 —— 这正是
protocol.ts:4875自己写下的规则。复用 authz bridge 的形状意味着一种模式而非两种。peer 从自己的 DB 读回而不是信任 payload,让 DB 保持唯一真相源。 - 防 AI 写代码犯错:⭐ 这一面最不含糊。B 恰恰是那种消费侧容忍,会让 AI 创作出来的对象「半存在」而平台替它遮掩,并且它削弱的是一道 fail-closed 门(ADR-0005 / 曝露 gate 对未知对象放行所依赖的「data path 会 404」并不成立 —— findData 无存在性校验(#3545 同类前提) #3770 / org 作用域的 flow overlay 只在「本进程内发布后」绑定触发器,重启后静默失绑——冷启动两条读路径都把 organization_id 非空的行滤掉了 #6190),那道门的关闭姿态是多租户正确性的承重件,不是整洁问题。
- 创业阶段不扩散需求:A 修的是已发货路径上的 p0,不加投机面,新契约是「一个频道 + 一个 attach 方法」,且照着一个已经发货的样子做。C 最大且与 plugin-reports: the documented setInterval dispatcher fallback is unreachable on ObjectKernel, and the job fallback it takes instead never fires #10746「fallback 不得伪装能力」的裁定相撞。
⇒ 建议 A。 若 EE 现网需要止血,B 可以作为明确临时、大声记录的补充与 A 并行 —— ⛔ 但不能当作修复本身。
⭐ Fold in #13609 — one fix closes both
#13609 is open and
needs-user-decisionfor the same seam from the delete side ("MetadataManager's unregister DOES broadcast — the observed prolongation sits in an unidentified seam"). This reading identifies that seam. ⇒ Whoever takes the decision should take both; splitting them means designing the same channel twice.Findings filed on the way past
- [finding]
MetadataFacade's class docblock claims it is "Registered as the 'metadata' kernel service" — the same file says 60 lines later that nothing onmaininstalls it #14019 —MetadataFacade's class docblock claims it is "Registered as the metadata kernel service" while the same file at:134says nothing onmaininstalls it. ⭐ Very likely the source of this card's inverted premise. - [finding]
MetadataClusterBridgePluginlogs "bridged metadata.changed → cluster.pubsub" on the in-processmemorydriver, which fans out to nobody — its siblingAuthzClusterBridgePluginguards exactly this, and the metadata bridge has no test file at all #14021 —MetadataClusterBridgePluginlogs "bridged" on the in-process memory driver that fans out to nobody, where its siblingAuthzClusterBridgePluginguards exactly that; and it has no test file at all.
Generated by Claude Code
- Two-arm kernel experiment, direction declared in writing first. Arm A (host-config shape):
7 remaining items
Claim: os-dev implementation seat, dispatched by PM loop round 11 (its claim comment is 5491366749)
Session:session_01Q5WBDtaUnoz5XuJ6jk8pQ5·mode:subagentos-dev (the dispatching PM seat's subagent; the branch below is this claim's identity bit)
Branch:claude/issue-13331-metadata-registry-cluster-fanout(pushed empty as the write-route probe before any edit)
Worktree:objectstack-issue-13331
File surface:packages/metadata-protocol/src/**(expected primary) pluspackages/services/service-cluster/src/**(bridge). READ-ONLY fences honoured:packages/objectql/src/engine.ts(held by PR #13864) andpackages/spec/**(single owner, not this lane) — stop-and-report on breach, no writes there under any circumstance.
Ruling being implemented: Option A per maintainer 2026-09-01 director batch A (comment 5486837547) — publisher at the metadata-protocol's onMetadataMutation choke point, service-cluster bridge late-binds at kernel:ready, peers converge from their own sys_metadata read; Option B not built.
Generated by Claude Code
- added a commit that references this issue
on Sep 1, 2026 os-dev-report
{ "issue": 13331, "status": "done", "branch": "claude/issue-13331-metadata-registry-cluster-fanout", "pr": "https://github.com/objectstack-ai/objectstack/pull/14183", "premise_still_valid": true, "clause_2": "yes — content limb as ruled (new public cross-node contract: channel METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated', address-only payload ClusterMetadataMutationPayload, attach/detach entry point attachMetadataMutationPubSub on the protocol). Path limb: NO, measured from the actual diff — packages/spec untouched, packages/objectql/src/engine.ts untouched. needs:contract-review attached at PR creation (additive REST POST; response echoed the label; re-read at report time pending the delayed-confirm discipline).", "zone2_verdicts": { "1_state_owner_is_protocol": "CONFIRMED — the registry write-through choke point AND the sys_metadata read the ruled receipt needs both live in ObjectStackProtocolImplementation; the engine precedent (attachAuthzInvalidationPubSub, engine.ts:3298 read-only) was mirrored in shape only. engine.ts has zero writes; the card proceeded this round.", "2_onMetadataMutation_choke_point": "CONFIRMED by symbol on this ref: onMetadataMutation at protocol.ts:4899, docblock 'the ONE choke point every authoring surface funnels through' at :4887 (drifted from the cited ~4875 — verified by symbol as instructed). PARTIAL NUANCE, measured: the funnel covers saveMetaItem / runPublishSideEffects / deleteMetaItem, but the two recovery doors (rollbackMetaItem, revertCommit) run the registry write-through WITHOUT emitting the event — pre-existing gap, local listeners miss it too; filed as #14179 with Blocked-by on this card.", "3_extend_existing_bridge": "CONFIRMED — MetadataClusterBridgePlugin gained an independent lane 2 (protocol service, duck-typed) beside the untouched lane 1; Runtime's default composition already mounts this plugin, so no new wiring anywhere. Lane 2 carries the isInProcessClusterDriver guard from birth; #14021's lane-1 misreading NOT absorbed (lane 1 byte-identical behavior, warn line verbatim).", "4_reuse_applyRegistryWriteThrough": "CONFIRMED — the peer applier is its deliberate FIFTH caller (active row present), with restoreArtifactRegistryView for the no-active-row leg; the caller-count pin, header trace and a route-5 wire-plural fold case are updated in protocol.object-registry-write-through-spelling.test.ts." }, "spec_fence_answer": "No split owed — the payload lives entirely in metadata-protocol types, matching both shipped precedents (authz payload in core's channel module; ClusterMetadataChangedPayload in the metadata package); IPubSub from spec/contracts is the existing generic transport. The spec's dormant MetadataChangedEventPayloadSchema: zero producers, zero consumers in-tree, declared version-compare semantics opposite to the ruled re-read receipt, z.bigint() version that cannot cross JSON — filed as #14180 for the spec lane.", "summary": "Option A implemented as ruled. The protocol's post-persistence funnel now publishes the mutation's address on the new metadata.mutated cluster channel (drafts never published, originNode loopback suppression, partitionKey type:name); on receipt a peer re-reads the row from its OWN sys_metadata and re-runs the registry write-through (or the delete heal walk when no active row remains), then replays the event into its local onMetadataMutation listeners without re-publishing — authored hook/action rebinds converge on peers too. The bridge late-binds the new seam at kernel:ready as an independent second lane, so the shipped EE boot shape (host-config, fallback metadata slot, lane 1 warns) gets the data-plane fix. Option B not built. Both fences held: engine.ts and packages/spec have zero writes.", "files_changed": [ ".changeset/metadata-mutation-cluster-fanout.md", "packages/metadata-protocol/src/index.ts", "packages/metadata-protocol/src/protocol.ts", "packages/metadata-protocol/src/protocol.cluster-mutation-fanout.test.ts (new)", "packages/metadata-protocol/src/protocol.object-registry-write-through-spelling.test.ts", "packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts", "packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts (new — first test file for this plugin)", "scripts/engine-double-contract.pinned.json (gate's own --write)" ], "tests": "TWO-ARM, directions declared in the test-file header before running: Arm B (bridge attached, two protocol replicas over one shared store) — peer converges from its own read, 'Arm B' case asserts peer registeredObjects == [gadget]; Arm A (control, no attach) — identical write leaves peer empty, so Arm B is the bridge's doing. 11 cases total in protocol.cluster-mutation-fanout.test.ts: address-only payload key-set, draft silence, loopback, duplicate-delivery convergence, re-attach idempotency, detach, delete fan-out heal (unregisterObject + removeObjectOverlay on the peer), draft-discard keeping the peer's active registration, listener replay ordered after registry convergence. COMMITTED-TREE ABLATION at 44a2e59b, direction declared first: deleting the single publisher call — mutation proved ON DISK by anchored greps (anchor count 0, marker count 1), not by editor exit codes — went red on exactly the 7 declared bus-crossing cases, 15 green (Arm A, loopback, draft silence, idempotency, route-5 all held); restore proved by state: HEAD-blob equality d7ba0225 both sides, empty git diff HEAD, marker count 0; restored run 22/22 green; trap EXIT INT TERM restore with absolute REPO_ROOT paths was armed but the state checks are the proof. REBUILD NOTE: these tests import ./protocol.js relative from the same package src — the subject never resolves through dist/exports, so no build leg exists on either ablation leg; stated rather than skipped. SUITES at final head e4601935: metadata-protocol vitest 'Test Files 152 passed | 2 skipped (154), Tests 2094 passed | 10 skipped'; service-cluster 'Test Files 6 passed (6), Tests 79 passed (79)'; tsc --noEmit clean, with both edited test files confirmed IN the program via --listFiles (2 hits — the exclude trap checked, not assumed). All heavy runs through scripts/pm/os-verify-lock.sh; verdicts read from its VERDICT lines (command-exit 0), never bare dollar-question.", "gates": "Derived at the ACTUAL diff, no paths passed: node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands; provenance line names objectstack-ai/objectstack at e4601935 and the --repo assertion holds. 44 commands derived (grew from 37 after the ledger touch pulled scripts-family gates — re-derived and re-run at final head). 41 exit 0 (exit captured by redirect before any pipe, per-gate logs kept). 3 NOT MEASURED by their own printed verdicts, never read as pass or fail: check-test-completeness (exit 3, grades a CI turbo log none exists locally), check:dual-build-cjs-loads (exit 3, needs full workspace build — CI-owned), check:type-check-debt (exit 3, full re-measure — CI-owned). Two first-pass reds fixed and re-run to their own OK lines: check:engine-double-contract 'OK — 741 pinned, 134 in the DEBT ledger, 3 exempt' (ledger learned the fanout file's pinned doubles via the gate's own --write); check:objectql-double-limit 'conformance holds: 299 doubles graded' (the new find double now holds the caller's bound by presence). check-nul-bytes OK (7740 files). Repo-level lint scan not run locally — CI-owned per standing rule.", "mcp_calls": "0 — card body + all 9 comments read via the zero-quota page payload (first 4) then repo-scoped REST after a green probe (HTTP 200 on /issues/13331); all writes (claim comment 5491428456, findings #14179/#14180/#14181, draft PR #14183, needs:contract-review label, this report) via the REST core bucket; dedup via REST open-issues list (5 pages, 443 issues, completeness reconciled against open_issues_count 452 incl PRs) + local grep with a firing positive control (attachClusterPubSub: 3 hits incl #13331).", "deviations": [ "Channel switch, declared: the page-payload archive carried only 4 of 9 comments (SSR backTimelineItems ships empty; timeline totalCount 36) — switched to repo-scoped REST after the probe answered 200. No comment or body showed sanitizer truncation (every tail ends in its footer).", "Receipt behavior beyond the ruling's minimum, declared in the PR for contract review: after registry convergence the peer replays the event into its LOCAL onMetadataMutation listeners (never re-published) — the remote-replay shape MetadataManager.attachClusterPubSub already ships for watchers; it is what makes authored hook/action rebinds converge on peers.", "The applyRegistryWriteThrough caller-count pin was re-opened 4 to 5 — the pin's own designed procedure (human re-reads the trace): trace updated, route-5 spelling case added, fold via canonicalMetaType at the wire boundary.", "PR body footer: platform appended a second bare footer after my session-URL footer at create time; left as-is per create-only treatment (no re-edit loop).", "origin/main advanced twice during the run (2bff79ae3 to 45b90512, PR #13864 landed among others); measured ZERO file intersection with my merge-base diff, so no merge was performed — CI tests the merge result. The anticipated #14114 barrel conflict cannot arise: no export was added to service-cluster's index.ts." ], "out_of_scope_findings": [ "filed as #14179: rollbackMetaItem / revertCommit run the registry write-through but never emit onMetadataMutation — local boot-cached rebinds and (post-#13331) cluster fan-out both miss recovery writes; Blocked-by: #13331", "filed as #14180: spec's MetadataChangedEventPayloadSchema declares a MUST-emit metadata persistence contract with zero producers/consumers in-tree and a z.bigint() version that cannot cross JSON — ADR-0049 enforce-or-remove candidate, spec lane", "filed as #14181: service-cluster tests compile under no tsc program (services-family instance of #14062's class) — a pre-existing TS2322 sits in memory.contract.test.ts, the package's contract witness" ], "open_questions": [] }
Generated by Claude Code
os-dev-report
{ "issue": 13331, "status": "done", "round": "rework — CI census red on PR #14183, repaired and validated", "branch": "claude/issue-13331-metadata-registry-cluster-fanout", "pr": "https://github.com/objectstack-ai/objectstack/pull/14183", "premise_still_valid": true, "clause_2": "restated as instructed: content limb YES per the ruling (new public cross-node contract — channel 'metadata.mutated', address-only payload ClusterMetadataMutationPayload with originNode, attach/detach entry point on the protocol); path limb NO, confirmed from the landed diff — ClusterMetadataMutationPayload lives in packages/metadata-protocol, packages/spec has zero writes, packages/objectql/src/engine.ts has zero writes. The spec fence held; no contract-first split is owed.", "summary": "check-system-context-census red at e4601935 was pure line rot from this PR's own insertion: the metadata.mutated publisher block sits above stripReadonlyForInsert in protocol.ts, moving its context.isSystem read 1737 to 1741 while content/docs/permissions/system-context.mdx row 21 still anchored 1737. Reproduced locally byte-for-byte (same two problems, exit 1 captured by redirect), repaired with the gate's own --fix, and VERIFIED as a SHIFT not a population change: the tool rewrote exactly one line (the anchor's line number), zero prose, and did not refuse; census population identical before and after (109 sites / 145 anchors). Bare re-run green: 'OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read.' Committed as cdbe5f0b, pushed after validation — one validated push. PR body notes the docs file as the re-anchor, not scope creep.", "gates": "Union re-derived at cdbe5f0b (provenance line names this repo at that commit; --repo holds): the docs file grew the family 44 to 70. All 70 run at cdbe5f0b: 68 exit 0 — including check-system-context-census green, and check:skill-examples upgraded from a prerequisite refusal to a REAL green ('260 prose examples type-check across 3 surfaces') after building the client-SDK closure it reads (rest, runtime, plugin-auth, verify, client, client-react; two closure quirks en route, neither mine: runtime's dts needs @objectstack/rest dist via a dev-edge outside the dependency closure, and verify races runtime's dts write under workspace-concurrency 2 — resolved by serial builds). 2 remain NOT MEASURED by their own printed verdicts, CI-owned: check-test-completeness (exit 3, grades a CI turbo log) and check:dual-build-cjs-loads (exit 3, needs the full 47-package workspace build). Suites re-anchored at cdbe5f0b: fanout+spelling+mutation-listeners 36/36, service-cluster 79/79 (VERDICT command-exit 0 under the verify lock); the full metadata-protocol suite's 2094-test green stands cited at e4601935 — the only delta to cdbe5f0b is one line number in a docs table, no package source touched.", "docs_drift_assessment": "content/docs/kernel/cluster.mdx section 6.2 is now INCOMPLETE, as the coordinator suspected: it states cross-node metadata invalidation 'already works' through the manager's metadata.changed lane alone — exactly the account #13331 falsified for the runtime-authoring path (the protocol never reaches the manager; host-config boots carry no manager at all). It needs a paragraph on the metadata.mutated protocol lane and the bridge's second attach. Its own closing principle there ('the peer re-reads the shared store, which is the source of truth') is the rule the new receipt implements, and its 'Target spec (planned)' block already marks MetadataChangedEventPayloadSchema as not wired — corroborating finding #14180. Same incompleteness class for the section 5/7 bridge mentions and content/docs/concepts/metadata-lifecycle.mdx. Stated in the PR body and here; corpus NOT rewritten per dispatch; releases pages untouched.", "files_changed_delta": [ "content/docs/permissions/system-context.mdx (one line: anchor 1737 to 1741 — the census gate's own --fix; noted in the PR body)" ], "mcp_calls": "0 this round (cumulative 0) — repro, fix, gates, push all local/git; PR body PATCH and read-backs via repo-scoped REST (PATCH sent body field only; draft read back true before and after).", "deviations": [ "check:skill-examples' local reading required building six packages outside my change's own closure — done under the verify lock rather than recorded NOT MEASURED, because the gate's refusal text names the exact prerequisite and it was locally satisfiable; the two remaining exit-3 gates are not (CI-owned inputs).", "PR body PATCH: the session-URL footer survived this PATCH (both footers present at tail); no re-paste loop performed either way." ], "out_of_scope_findings": [], "open_questions": [] }
Generated by Claude Code
os-dev-report
{ "issue": 13331, "status": "done", "round": "merge-conflict resolution on PR #14183 — resolved with the tooling, one validated push", "branch": "claude/issue-13331-metadata-registry-cluster-fanout", "pr": "https://github.com/objectstack-ai/objectstack/pull/14183", "premise_still_valid": true, "summary": "git merge origin/main AUTO-completed (58321d2d) — no textual conflict locally — but the auto-merged census page was WRONG: it kept this branch's pre-#14171 anchor rows beside my row-21 edit, and the bare census gate flunked the merged tree 26 ways (paired stale-anchor / unanchored-site entries in engine.ts, deltas +64..+95, and share-link-service.ts, delta +11 — the coordinator's predicted mechanism, adjacent rows of one machine-maintained table). Resolved exactly per instruction: took origin/main's page wholesale, re-derived with check-system-context-census --fix on the merged tree. The tool performed a SHIFT, not a population change, and did not refuse: it rewrote exactly ONE anchor — row 21, protocol.ts 1737 to 1741 (re-derived on the merged tree; same number as before but not carried forward on assumption). Delta vs origin/main's page is that single line, zero prose. The repo's own os-regen commit hook confirmed the deferred generated artifact current and cleared its marker. Committed as 0cf0a437, pushed once after full re-validation. PR now reads mergeable true / mergeable_state blocked (normal draft-plus-pending-checks aggregate), head 0cf0a437, draft still true — not flipped ready, no enqueue, no auto-merge.", "readings_retaken_at_0cf0a437": [ "census gate bare: 'OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read' (red 26 problems before resolution, green after — both readings on the merged tree)", "metadata-protocol full suite: Test Files 152 passed | 2 skipped (154), Tests 2094 passed | 10 skipped — after rebuilding the dependency closures on the merged tree (main's five commits touch packages/spec sources, so pre-merge dists were stale; 判断任何事之前先 build)", "service-cluster full suite: Test Files 6 passed (6), Tests 79 passed (79)", "metadata-protocol tsc --noEmit: clean (same lock hold, VERDICT command-exit 0)", "check-nul-bytes: OK (7757 files)" ], "readings_carried_forward": "Everything else is carried, stated as such and not restated as fresh: the 70-gate union (68 green / 2 CI-owned NOT MEASURED) stands at cdbe5f0b; the two-arm experiment and the committed-tree publisher ablation stand at 44a2e59b; the full-suite green also previously taken at e4601935/cdbe5f0b is superseded by the fresh 0cf0a437 run above. Per the coordinator, the union was not re-run — the merge brings no new files into this PR's diff (the only diff-file the five main commits touched is the census page itself, now resolved to main-plus-one-line).", "files_changed_delta": [ "merge commit 58321d2d (origin/main into the branch — no rebase, no force-push, no stash)", "0cf0a437: content/docs/permissions/system-context.mdx re-derived on the merged tree (9 lines vs the merge commit — 8 rows restored to main's re-anchors, 1 row re-shifted for this branch's insertion; exactly 1 line vs origin/main)" ], "mcp_calls": "0 this round (cumulative 0) — merge, resolution, validation, push all local/git; PR state read via repo-scoped REST.", "deviations": [ "The coordinator's step 1 anticipated a conflicted merge; git auto-merged instead — but the auto-merge was semantically wrong in exactly the predicted file, caught by running the census gate on the merged tree BEFORE any push. Steps 2-4 were then followed verbatim (main's page wholesale, --fix re-derive, shift verified, no refusal).", "Resolution was committed as its own commit (0cf0a437) after the merge commit rather than folded into it: the census tool was run against a committed merge tree, mirroring the repo's commit-then-regenerate rule for generated artifacts; the repo's os-regen hook endorsed the sequence (deferred-artifact marker cleared at commit)." ], "out_of_scope_findings": [], "open_questions": [] }
Generated by Claude Code
os-support-ai commented
on Sep 1, 2026 CollaboratorMore actions✅ Contract review PASS at
CONTRACT_REVIEW_TIER→needs:contract-reviewcleared, PR #14183 ready and enqueueddomain:enginelane PM, sessionsession_01Q5WBDtaUnoz5XuJ6jk8pQ5, R11.⛔ This seat is below tier, so it did not self-review. Clause-② was ruled to fire on this card, and the enqueue gate forbids landing below
CONTRACT_REVIEW_TIER— so the review was delegated to a reviewer running at that tier, per the in-seat review rule. Its full verdict is on the PR at5494333461. Verdict: PASS, no ESCALATE, three findings, all non-blocking.What the review measured (its readings, not mine)
- Content limb: YES, declared honestly and completely. The minted surface is exactly what the PR declares —
METADATA_MUTATION_CLUSTER_CHANNEL = 'metadata.mutated'andClusterMetadataMutationPayload(the only two new barrel exports) plusattachMetadataMutationPubSub/detachMetadataMutationPubSub. Base-tree grep at9dd022aefor all three: zero hits, with a firing positive control. No undeclared widening. - Two lanes are correct, not duplicate — measured: lane 1's only production publisher is
MetadataManager.notifyWatchers, andprotocol.tsat head has zero code references to the metadata service (the 3 hits are doc comments this PR adds). Host-config boots carry no manager at all, which is why lane 2 duck-types the real implementation. - Address-only verified at three levels: the payload type, all three emit sites, and the receiver — which consumes the wire purely as an address and registers from its own DB read. That is the ruling's load-bearing clause and it holds.
- Failure directions safe: duplicates idempotent (pinned), out-of-order converges because the DB read decides, a lost signal degrades to the pre-existing boot-reload bound, a failed apply fails closed (404 staleness, never wrong data).
- Option B was not built, as ruled.
What I verified myself before enqueueing (⛔ not taken from the review)
Check Reading ⛔ packages/objectql/src/engine.tsfenceHELD — 0 in the diff's file list ⛔ packages/spec/**fenceHELD — 0; no contract-first split owed Governed surface 0 across all 9 files ⇒ ready→enqueue path, not the maintainer fork Mergeable vs current main(b992b1d6)git merge-tree --write-treeexit 0, clean — measured locally rather than trusting the API's cached flag, sincemainmoved 8 commits after this head was built⚠️ Generated ledgerscripts/engine-double-contract.pinned.json0 commits touched it since the merge base (8 landed overall; control confirms the path is tracked with 2 prior commits) ⇒ no silent-drop risk, so the regen four-step is not owed here CI at head 0cf0a43733 checks, 32 success + 1 skipped, zero failures ⭐ The ledger check is the one I would not have skipped: a clean textual merge on a generated artifact is exactly where one side's entries vanish without a conflict — this lane paid for that lesson earlier today on the census page. Here it was genuinely clean, and measured rather than assumed.
The three findings, and their dispositions — all non-blocking
- Peer apply race — no per-key serialisation at the receiver, so two rapid same-row mutations can under DB-response jitter leave a peer one version behind until the next signal or boot. Same class as two racing local writers, and inside the PR's own declared staleness bound. Receiver-internal, fixable later with no contract change ⇒ not a landing blocker.
deleteMetaItem's legacy control-plane branch never emitted even locally (pre-existing, marked deliberate), so it fans out to no one. Same family as metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 but not named there ⇒ recommendation is to append it to metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179, ⛔ not widen this PR.- Wording — the payload docblock's "at-least-once … harmless" is a tolerance statement, not a delivery guarantee. The changeset and PR body already state the at-most-once loss bound plainly.
⛔ None of the three is a design fork the ruling leaves unsettled, so none goes to the maintainer.
State transition
needs:contract-reviewcleared (read back: the other four labels intact, nothing else stripped) · draft → ready · auto-merge enabled, in the merge queue.⚠️ Owed at landing — do not let this one slip#13609 is
pm:blockedon this card per the ruling, and the ruling is explicit: when this lands, re-verify the deleted-entry prolongation on a multi-node deployment before closing it. ⭐ That is not a formality — this PR's own reading is what identified the seam #13609 observed from the delete side, so #13609 is the check that the identification was right. I will carry it at the landing.Also queued behind this landing: #14179 (recovery doors
rollbackMetaItem/revertCommitrun the write-through but never emit the mutation event even locally) carriesBlocked-bythis card, and finding 2 above belongs on it.
Generated by Claude Code
- Content limb: YES, declared honestly and completely. The minted surface is exactly what the PR declares —
os-dev-report
{ "issue": 13331, "status": "done", "round": "second merge-conflict resolution on PR #14183 — mergeable again, one validated push", "branch": "claude/issue-13331-metadata-registry-cluster-fanout", "pr": "https://github.com/objectstack-ai/objectstack/pull/14183", "premise_still_valid": true, "summary": "git merge origin/main auto-completed again (d2e4679a); per the known trap, textual cleanliness was treated as evidence of nothing. Followed the procedure verbatim: took main's content/docs/permissions/system-context.mdx wholesale (main's #14199 re-anchored 11 rows of that machine-maintained table), then re-derived on the merged tree with check-system-context-census --fix. The tool performed a SHIFT, not a population change, and did not refuse: exactly ONE anchor rewritten — row 21, protocol.ts 1737 to 1741. The number equals last round's but was RE-DERIVED, not carried: measured zero main-side commits on packages/metadata-protocol/src/protocol.ts since 0cf0a437, so this branch's insertion shift is unchanged — the differing-anchor warning was prudent and the measurement answered it. Bare gate green: 'OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read.' Delta vs origin/main's page is that single line, zero prose. The os-regen commit hook again confirmed the deferred artifact current. Committed 7ffeb726, pushed once after full re-validation. PR now: mergeable true, mergeable_state blocked (ready PR awaiting checks — auto-merge armed by the coordinator), head 7ffeb726, draft false — draft state and auto-merge untouched by me as instructed.", "pinned_ledger_recheck": "scripts/engine-double-contract.pinned.json re-measured after the merge as instructed: ZERO main-side commits touched it (git rev-list count over 0cf0a437..origin/main), and the gate run on the merged tree prints its own OK — 'check-engine-double-contract: OK — 741 pinned, 134 in the DEBT ledger, 3 exempt' — so the merge dropped nothing and no --write regeneration was owed.", "readings_retaken_at_7ffeb726": [ "census gate bare: OK line quoted above (on the merged tree, after --fix)", "check:engine-double-contract: OK — 741 pinned, 134 DEBT, 3 exempt (merged tree)", "metadata-protocol full suite: Test Files 152 passed | 2 skipped (154), Tests 2094 passed | 10 skipped — after rebuilding both dependency closures on the merged tree (main moved ~15 commits incl. spec/objectql/runtime sources)", "service-cluster full suite: Test Files 7 passed (7), Tests 93 passed (93) — the package GREW under the merge (#14114's multi-node-gate-mount.test.ts landed) and passes alongside the bridge tests; the two features coexist with no barrel conflict, as predicted", "metadata-protocol tsc --noEmit: clean (same lock hold, VERDICT command-exit 0)", "check-nul-bytes: OK (7769 files)" ], "readings_carried_forward": "Stated as carried, not fresh: the 70-gate union (68 green / 2 CI-owned NOT MEASURED) stands at cdbe5f0b; the two-arm experiment and committed-tree publisher ablation stand at 44a2e59b. The union was not re-run per the coordinator's earlier scoping — this merge brings no new files into the PR's diff beyond the census page already handled.", "files_changed_delta": [ "merge commit d2e4679a (origin/main into the branch — no rebase, no force-push, no stash)", "7ffeb726: content/docs/permissions/system-context.mdx re-derived on the merged tree (11 lines vs the merge commit — main's #14199 re-anchors restored, row 21 re-shifted; exactly 1 line vs origin/main)" ], "mcp_calls": "0 this round (cumulative 0) — merge, resolution, validation, push all local/git; PR state read via repo-scoped REST.", "deviations": [ "The merge again auto-completed rather than conflicting; the semantic wrongness was assumed rather than trusted-away, and the census gate was run on the merged tree before any push — red-to-green sequence observed locally (the --fix rewrite is itself the red's proof: main's page anchored 1737 for a tree where the read sits at 1741)." ], "out_of_scope_findings": [], "open_questions": [] }
Generated by Claude Code
os-support-ai commented
on Sep 1, 2026 CollaboratorMore actionsConflict resolved — PR #14183 mergeable again at
7ffeb726. Contract-review PASS and auto-merge untouched.domain:enginelane PM, sessionsession_01Q5WBDtaUnoz5XuJ6jk8pQ5. Maintainer flagged the conflict; routed to the branch's own dev rather than a second actor on the branch.⚠️ The conflict was invisible to every textual check — againgit mergeauto-completed with no conflict, for the second time on this branch. ⛔ That was treated as evidence of nothing, which is the only correct reading here: main's #14199 (34ce8e7d) re-anchored 11 rows ofcontent/docs/permissions/system-context.mdxwhile this branch re-anchored 1. Different rows, so git merges them happily — but both sides' line numbers were derived against differentprotocol.tstrees, and the merged tree matches neither.⭐ The
--fixrewrite is itself the proof the merge was wrong: main's page anchored 1737 for a tree where the read sits at 1741.Resolution per procedure: main's page taken wholesale, re-derived with
check-system-context-census --fixon the merged tree. A shift, not a population change — exactly one anchor rewritten, zero prose. Bare gate green: "OK — 109 elevation read sites in 20 packages across 45 files, all anchored; 145 anchors resolve, 27 declared non-read."⭐ And the number that came back was the same as last round — which the dev measured rather than waved through. I warned to expect a different anchor and not to carry
1741forward. It re-derived, got1741again, and then answered why: zero main-side commits touchedpackages/metadata-protocol/src/protocol.tssince0cf0a437, so this branch's own insertion shift is unchanged. A matching number reached by measurement is a reading; the same number carried forward would not have been.The two things I asked to be re-measured after the merge
- Generated ledger
scripts/engine-double-contract.pinned.json— re-measured on the new merge base: zero main-side commits touched it, and the gate prints its own OK on the merged tree (741 pinned, 134 DEBT, 3 exempt). Nothing dropped; no--writeregeneration owed. My pre-enqueue reading was 15 commits stale, so this was the right thing to re-take rather than inherit. - Suites on the merged tree —
metadata-protocol2094 passed,tsc --noEmitclean, after rebuilding both dependency closures (main moved ~15 commits including spec/objectql/runtime sources).
⭐ A predicted risk resolved itself, and the measurement says so
service-clustergrew under the merge — 7 files / 93 tests, up from 6 / 79 — because #14114'smulti-node-gate-mount.test.tslanded. That is the PR I named at dispatch as the barrel-conflict risk, with the standing ⛔ never to resolve such a conflict by dropping its exports. It never arose: this card added no barrel export, the two features coexist, and both suites pass together.State
mergeable: true, ready, auto-merge still armed from the contract-review pass — the dev left draft state and auto-merge untouched as instructed. It will re-queue and land on its own once CI is green on7ffeb726. ⛔ Nothing about the review outcome was reopened.📌 Still owed at landing: #13609's multi-node re-verification of the deleted-entry prolongation, per the ruling.
Generated by Claude Code
- Generated ledger
os-support-ai commented
on Sep 1, 2026 CollaboratorMore actions✅ LANDED — PR #14183 merged as
1403d943, verified by content onorigin/mainpm:dispatched→ stripped.pm:blocking→ stripped (both blockees released below). The lane's onlypriority:p0is closed.The landing probe, and why it discriminates
⛔ Per this seat's own rule, the API's
mergedfield is not the evidence. The probe was run against the tree, and it had to be able to come back negative:parent 1134e47aorigin/main1403d943METADATA_MUTATION_CLUSTER_CHANNEL|ClusterMetadataMutationPayload|attachMetadataMutationPubSubacrosspackages/absent (exit 1, empty) protocol.ts×7,index.ts×1,metadata-cluster-bridge-plugin.ts×4, plus two new test files⭐ Positive control on the same parent commit, same instrument, same path: a control string in
protocol.tsreturned 4 on1134e47a. So the zero above is a reading — the probe was pointed at a file that existed and was being read — not a mis-spelled query against a tree the grep never reached.Independently:
git log origin/main -- <file>names1403d943as the last touching commit for both subject files (packages/metadata-protocol/src/protocol.ts,packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts).Fences — both held post-merge
Declared at dispatch as untouchable; re-checked against the merged tree rather than against the pre-merge diff:
packages/spec— not in the merge's changed-file list; last touched by feat(spec): the release artifact may carry N package manifests — optionalpackages[](ADR-0130 D4) #14191 and feat(spec,objectql,runtime): declare ctx.referentialFieldClear on HookContextSchema, populate every set_null cleanup write, and carry it across the QuickJS sandbox boundary #14199, neither of them this PR.packages/objectql/src/engine.ts— not in the list; last touched by feat(spec,objectql,runtime): declare ctx.referentialFieldClear on HookContextSchema, populate every set_null cleanup write, and carry it across the QuickJS sandbox boundary #14199.
What actually landed (9 files, +1199 / −45)
packages/metadata-protocol/src/protocol.ts +245 packages/metadata-protocol/src/protocol.cluster-mutation-fanout.test.ts +466 (new) packages/metadata-protocol/src/protocol.object-registry-write-through-spelling.test.ts +61/− packages/metadata-protocol/src/index.ts +7 packages/services/service-cluster/src/metadata-cluster-bridge-plugin.ts +182/−45 packages/services/service-cluster/src/metadata-cluster-bridge-plugin.test.ts +214 (new) .changeset/metadata-mutation-cluster-fanout.md +52 scripts/engine-double-contract.pinned.json +15 content/docs/permissions/system-context.mdx ±1⚠️ Two of these are the hazard classes this seat has been tracking, and both came through intact:content/docs/permissions/system-context.mdx— the census anchor table, ±1 line. This is the file that merges cleanly and lands semantically wrong when two PRs derive their anchors against differentprotocol.tstrees. It was re-derived with the gate's own--fixon the merged tree, not hand-edited.scripts/engine-double-contract.pinned.json— a generated ledger, where a zero-conflict merge silently drops one side. Concurrent-commit count on that path was measured twice: before enqueue and after the merge, zero both times.
Emit-site inventory on the landed code
emitMetadataMutationhas exactly three call sites, one per authoring verb —saveMetaItem(:15071),publishMetaItem(:16305),deleteMetaItem(:20043) — with the local half (emitMetadataMutation→ listeners) and the cluster half (publishMetadataMutation→METADATA_MUTATION_CLUSTER_CHANNEL) split so a received peer event replays locally without re-publishing. That split is the loop guard;originNodeon the payload is the second.📌 Released by this landing
Both were
pm:blockedon this card and both move topm:queuein this same pass:- [finding] QA observed /api/v1/meta/datasource serving a deleted datasource's entry cluster-wide, but MetadataManager's unregister DOES broadcast — the observed prolongation sits in an unidentified seam #13609 — the re-verification carrier per the maintainer's ruling (director batch A, 2026-09-01). ⭐ Its premise was re-measured against the landed code before unblocking, not assumed; see the note there.
- metadata-protocol: rollbackMetaItem / revertCommit registry write-throughs never emit onMetadataMutation — boot-cached rebinds and (post-#13331) cluster fan-out miss recovery writes #14179 — the recovery-door emit gap. A fourth non-emitting site of the same class was found on the landed tree and appended there.
Generated by Claude Code
- added a commit that references this issue
on Sep 1, 2026 - added a commit that references this issue
on Sep 2, 2026 - added a commit that references this issue
on Oct 7, 2026
Symptom
On a 3-replica objectos-ee cluster (framework pin
502ff8b5, redis cluster driver active — see #13330 for getting it active at all), every replica logs at boot:Measured consequence — create a new object through the load balancer, then immediately write data to it through the LB:
Only the replica that handled the metadata write registers the object with its ObjectQL engine; the other two keep answering
OBJECT_NOT_FOUNDindefinitely (no TTL/poll fallback on this path). Restarting the replicas is the only recovery.GET /api/v1/meta/*list/read stays fresh everywhere (store-backed reads), which makes the drift worse to spot: the metadata API says the object exists while the data API on two of three nodes refuses it.Related earlier fix #5109 addressed peer-side
listCache/registry invalidation once the broadcast arrives — here the broadcast never starts, because the bridge never attaches.Root cause direction
MetadataClusterBridgePlugin(service-cluster) resolvesctx.getService('metadata')atkernel:readyand requiresattachClusterPubSubon it (metadata-cluster-bridge-plugin.ts:53-59).NodeMetadataManager extends MetadataManagerdoes carry that method (metadata-manager.ts:2969), and@objectstack/metadata'sMetadataPluginregisters exactly that manager (plugin.ts:303).But on this boot (objectos-ee,
objectstack.config.tsviaos start→serve), the kernel's plugin lineup contains no MetadataPlugin at all (boot banner lists ObjectQL, PlatformObjects, Dispatcher, … — no MetadataPlugin, and plugin.ts:304's registration log line never prints). Whatever registers the'metadata'service on the TS-config path (runtime dispatcher/domain facade) does not exposeattachClusterPubSub, so the bridge warns and bails — exactly the "protocol omission → stale caches" casecontent/docs/kernel/cluster.mdx§1 predicts. serve.ts:2296+ composes an artifact-sourced MetadataPlugin only on artifact boots.Second-order effect, same neighborhood: a
record_changeflow saved at runtime (PUT /api/v1/meta/flow/…) is registered ([Registry] Registered flow: …) but its trigger only arms after a replica restart, and then only on that replica's local hooks — runtime-authored automation is effectively single-node too.Repro
cloud/apps/objectos-ee/docker-compose.ymlat 3 replicas with the redis driver active (needs the cli: serve's cluster-driver load registers into the CJS registry while the ESM Runtime reads the ESM one — OS_CLUSTER_DRIVER=redis silently downgrades to "not registered" (post-#10645) #13330 workaround).PUT /api/v1/meta/object/<new>through the LB.POST /api/v1/data/<new>through the LB — two thirds failOBJECT_NOT_FOUND, and keep failing.Expected
Whatever object the TS-config boot registers as the
'metadata'service either exposesattachClusterPubSub()(delegating to the real manager underneath), or the bridge knows how to reach the manager behind the facade — sometadata.changedfans out and #5109's peer-side invalidation actually runs. Until then, the EE multi-node compose (ADR-0018) is safe only for deployments whose metadata is fixed at deploy time and whose replicas restart together after any schema change.Found during a 3-replica EE cluster deployment verification (2026-08-30); companion report in cloud branch
claude/enterprise-cluster-deployment-qxiexe.