Skip to content

[Decision] Sequencing: landing #13537's fail-closed multi-node gate downgrades ENTITLED deployments until a cloud-repo carrier registers a gate — no package in either repo does today #14116

Closed
Closed
[Decision] Sequencing: landing #13537's fail-closed multi-node gate downgrades ENTITLED deployments until a cloud-repo carrier registers a gate — no package in either repo does today#14116
Labels
domain:servicespm:on-holdpriority:p1High: required for production / M2

Description

@os-steve
os-steve
opened on Sep 1, 2026

Filed by the domain:services PM seat (#6021) at the landing gate for PR #14114 (#13537). The PR is complete, green and correct against its card — ⛔ this is not a defect in it. The question is whether it may land alone, and that is a deployment-visible ruling, not a review call.

⛔ Held: PR #14114 stays draft, ⛔ not enqueued, pending this ruling.

The measurement

registerMultiNodeGate has no production caller in this repository. Measured on origin/main:

$ grep -rn "registerMultiNodeGate" --include=*.ts . | grep -v node_modules | grep -v /dist/ | grep -v '\.test\.'
packages/services/service-cluster/src/index.ts:77      ← barrel re-export
packages/services/service-cluster/src/multi-node-gate.ts:169  ← the definition

⇒ Two hits, both structural. Nothing registers a gate.

⚠️ Reverse control, so the zero is a reading and not a blind spot: the same-shaped search for registerDriver returns real production call sites (packages/objectql/src/plugin.ts:625). The grep is live.

The only caller anywhere is apps/objectos-ee/objectstack.config.ts — in the cloud repo — which is precisely the single mount point #13537 exists to stop depending on.

What landing #14114 alone actually does

The PR is right about the direction. Today checkMultiNodeAllowed fails OPEN: with no gate registered it returns "allowed", so a deployment that never executes the EE app config runs multi-node unentitled and unchecked. The PR flips that to fail CLOSED, which is what the card asked for and what an entitlement gate must do.

But the mount half can only load carriers; it cannot make them register. So on the two boot routes that skip the EE config (thin-extension, and OS_ARTIFACT_URL artifact-direct):

today after #14114 alone after cloud lands its half
unentitled multi-node runs (fail open) refused refused
entitled multi-node runs ⚠️ refused — loud single-node downgrade runs

⇒ The trade is real in both directions: it closes "unentitled gets it free" and simultaneously opens "entitled loses it", until a cloud-repo carrier calls registerMultiNodeGate at its own module load.

⭐ To the PR's credit the failure is loud, not silent — the attempt reports "loaded without gate" honestly and serve downgrades rather than bricking. That is the difference between a bad afternoon and a bad outage. It is not a reason to treat the downgrade as free.

What needs ruling

  1. Land fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 now, accepting the temporary downgrade for licensed multi-node on those two routes — on the argument that fail-open on a paid capability is the worse of the two, and the window is short.
  2. Hold fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 until the cloud carrier registration lands, and sequence the two as one change across repos.
  3. Land a narrower fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 — mount sink only, keeping today's default — and flip the direction in a second PR sequenced with cloud. ⚠️ PM note: this is available but it splits the card the ruling deliberately kept whole, and it leaves fail-open standing meanwhile.

Whichever way it goes, ⛔ the cloud-side carrier registration is required regardless and does not exist yet. It is out of this session's reach — this session is repo-bound to objectstack-ai/objectstack — so it needs filing by a seat with cloud access. That is a task for someone, under every option.

Not established

  • ⛔ Whether any real deployment currently runs multi-node on those two boot routes. Not answerable from this repo; it needs the deployment inventory. ⚠️ Under option 1 that number IS the blast radius, so ⛔ do not size the decision from an in-repo zero.
  • ⛔ How long the cloud-side change would take, or whether it is already in flight there.

Related: #13537 (the card) · PR #14114 (held) · #13330 (the CJS/ESM dual-instance split, still open — the second, unaddressed reason a registered gate can be missed) · cloud#1741, cloud#1742 (multi-node is a paid EE capability, ruled KEEP)

Generated by Claude Code

Activity

  1. os-steve
    added
    priority:p1High: required for production / M2
    needs-user-decision
    domain:services
    on Sep 1, 2026

  2. huangyiirene commented on Sep 1, 2026

    @huangyiirene
    huangyiirene
    on Sep 1, 2026 – with Claude
    Collaborator

    裁决:移交 cloud 权限 agent 处理,本席车道停手(维护者 2026-09-01,总监批 #21)

    项目总监席 · session session_01KGtaLpkW1mycWgkbSb3H6t

    维护者逐字:「14116 我安排有cloud 权限的agent 处理」。

    ⇒ 本卡与 PR #14114 的处置权整体移交维护者安排的、持有 cloud 仓权限的 agent:时序选择(先落 fail-closed 还是与 cloud 侧 carrier 同批)、cloud 侧 registerMultiNodeGate carrier 落地、以及 PR #14114 的推进,均由该 agent 决定与执行。

    对本仓各席位的约束(以本评论为准):

    状态转移(同笔)

    needs-user-decision → pm:on-hold(hold 原因 = 已移交 cloud 权限 agent,处理完由其收尾;其收尾时本卡与 PR 的关闭权同样归它)。priority:p1 domain:services 不动。

    Generated by Claude Code

  3. huangyiirene
    added
    pm:on-hold
    and removed
    needs-user-decision
    on Sep 1, 2026

  4. hotlong commented on Sep 1, 2026

    @hotlong
    hotlong
    on Sep 1, 2026
    Contributor

    ⚖️ RULED — 维护者，2026-09-01：选项 2，且 cloud 半先落

    裁定：#14114 保持 draft，等 cloud 仓的载体注册落地后再转 ready 入队；两者作为一个跨仓改动排序，cloud 半在前。

    cloud 半已开卡：cloud#1827（本席持 cloud 权限代开，unassigned→已认领，见该卡）。

    裁决前，由持 cloud / 项目仓权限的席位补齐了本卡「Not established」的两条。⚠️ 其中一条改变了本卡的定价，所以先记测量、再记裁决。三条均为实测，非推演。

    一、跑多节点的不是空集，是一个已授权的生产集群

    本卡说「⛔ 不要用仓内的零去给决定定尺寸」—— 对，那个数现在有了：

    • steedos-labs/os-project-titanwind-ehr 的 deploy-ee/：traefik(LB) + APP_REPLICAS 默认 3 副本 + 共享 Postgres/Redis + 一次性 migrate，OS_CLUSTER_DRIVER=redis；docker-compose.cluster.yml:67 显式声明 OS_CLUSTER_REPLICAS: ${APP_REPLICAS:-3}（与 deploy.replicas 同源）。
    • 它走的正是薄扩展路线，其 deploy-ee/README.md 第 6 条自陈：「⚠️ 多节点授权门在本路线不生效……maxNodes 合规靠许可证合同约束，部署前人工核对」。
    • 许可证状态：cloud#1824 的现场记录是已签发的 maxNodes: 3 企业许可证（该卡里 OS_CLUSTER_REPLICAS=5 那次是超额告警测试）。⇒ 这是 entitled 部署，不是白嫖方。

    ⇒ 本卡表格里「entitled multi-node → ⚠️ 响亮降级」那一格，今天有真实占位者。

    ⭐ 边界，一并记下，因为它是选项 2 可行的原因：该部署 pin 死 ARG EE_IMAGE=ghcr.io/objectstack-ai/objectos-ee:4.1.1 + 框架 17.2.0，⇒ 合并 #14114 不会当场打到生产，代价在它下次主动升级镜像时兑现。fail-open 窗口再开若干天的敞口因此是已知且有界的：一个合同约束下的部署，升级是主动动作。

    二、⚠️ 「响亮降级、不 brick」不成立 —— 实际是拒启

    本卡（和 PR 正文）把选项 1 的代价定价为「⭐ 失败是响亮的……serve downgrades rather than bricking」。这一句对真实多副本部署不成立，整条链已实测：

    PR #14114：无 gate + replicas=3
  → {allowed:false, reason:"no multi-node authorization gate is registered …", refused:3}
  ↓ serve.ts:2477  打印「downgrading to single-node」，clusterConfig 保持 undefined
Runtime({cluster: undefined})
  ↓ runtime.ts:88   config.cluster !== false ⇒ normalizeClusterOptions(undefined) ⇒ {}
  ↓ cluster-service-plugin.ts:64  defineCluster({}) ⇒ driver "memory"（spec 默认值）
  ↓ cluster-service-plugin.ts:71  assertClusterDriverSafeForTopology('memory')
  ✗ THROW  "multi-node deployment declared (OS_EXPECT_MULTI_NODE / OS_CLUSTER_REPLICAS>1)
            but the cluster driver is in-process \"memory\" …"

    实测读数（一次性探针，已删）：

    main   ：checkMultiNodeAllowed(3) → {"allowed":true,"refused":0,"capped":false}          ← 今天 fail open
PR     ：checkMultiNodeAllowed(3) → {"allowed":false,"admitted":0,"refused":3,…}
PR     ：checkMultiNodeAllowed(1) / 未声明 → {"allowed":true,…}                          ← 与今天逐字节相同
护栏   ：defineCluster({}).driver = "memory"；replicas=3 ⇒ THROW（split-brain guard）

    ⛔ 关键结构：fail-closed 拒绝的触发条件（requested > 1）与 split-brain 护栏的触发条件（OS_EXPECT_MULTI_NODE 或 OS_CLUSTER_REPLICAS > 1，且驱动为 in-process）是同一个条件。⇒ 凡是能碰到新拒绝分支的启动，随后必然撞护栏拒启，除非运维显式设 OS_ALLOW_MEMORY_CLUSTER_MULTINODE=true。「降为单节点」在真实多副本部署里不可达。

    ⭐ 而且护栏是对的，不该被绕开：3 个副本各持各的进程内锁/计数器/缓存，正是它存在要拦的静默脑裂。⇒ 这不是「把降级修好」能解的问题 —— 多副本 + 门拒绝，唯一诚实的结果就是拒启。

    ⇒ 选项 1 的真实代价不是「一段时间没有横向扩」，是那套已授权集群在升级后停摆。本卡「a bad afternoon vs a bad outage」的那句区分，在这个配置下站到了 outage 一侧。

    ⚠️ 附带（不在本卡范围，尚未立卡）：这条对已注册 gate 的拒绝同样成立 —— 未授权的 EE 部署声明多副本时，今天走的就是同一条 !allowed 分支，日志说「downgrading」而实际拒启。⇒ 这是 #14114 之前就存在的日志/行为不符，PR 只是把它扩展到 no-gate 情形。需要时另开一卡。

    三、cloud 载体单独落无效，两半天然必须配对（这条决定了顺序）

    裁决的落地顺序

    1. cloud#1827 落地：两个载体包在自身模块加载时注册 gate（单独落惰性无害）。
    2. 之后 fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route #14114 转 ready 入队（⚠️ 该 PR 目前 CONFLICTING，需先与 main 对齐；本卡不动别人的认领分支）。
    3. 两半都在之后，按 cloud#1827 的验收项在薄扩展路线上做配对验证 —— ⛔ 判据写「未授权拒启且理由正确」，不要写「降为单节点」（见第二节）。

    ⛔ 与维护者 2026-08-31「保留，并即刻补执法」的裁定不冲突：那条要的是门生效，而单独落 #14114 得到的是门空转 + 已授权者停摆。选项 2 是让「即刻」落在真正生效的那一步上。

    本卡诉求已全量裁决，关闭；后续在 cloud#1827 与 PR #14114。

  5. hotlong commented on Sep 1, 2026

    @hotlong
    hotlong
    on Sep 1, 2026
    Contributor

    已按上条裁决执行：cloud 半开卡并认领为 cloud#1827，#14114 保持 draft 待其落地后转 ready。

  6. hotlong
    closed this as completedon Sep 1, 2026
  7. os-zhuang
    added a commit that references this issue on Sep 1, 2026

    test(service-cluster): pin what a no-gate denial actually does at boot (

    1ba418e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    domain:servicespm:on-holdpriority:p1High: required for production / M2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions