Skip to content

fix: fail closed when model capabilities change from true to false #3608

Closed

Description

@M4n5ter
English

Problem

At PR #3565 exact head c658f3c15fcfc80fa0b16eb9467de471ee953d52, refresh:model-metadata does not treat a boolean capability changing from true to false as a projection removal.

scripts/sync-model-metadata.mjs recursively records removed object keys and array members, but collectProjectionRemovals() returns as soon as the previous value is a primitive. Consequently both of these supported upstream transitions currently publish without --accept-upstream-removals:

  • reasoning: true -> false
  • tool_call: true -> false, projected as capabilities.functionCalling: true -> false

A production-helper probe confirmed that each refresh completed and persisted the false capability. The existing eight focused tests all pass because they cover deleted models, pricing fields, object keys, and array members, but not boolean capability loss.

This contradicts the documented contract that a refresh fails closed when a committed capability disappears. It can remove a model from a tool-capable catalog or suppress reasoning controls without the explicit review/acknowledgment gate.

Acceptance criteria

  • Treat true -> false below a model's capabilities projection as a removal.
  • Keep ordinary scalar corrections, such as display names, status, limits, and prices, refreshable without the removal flag.
  • Preserve --accept-upstream-removals as the explicit acknowledgment path.
  • Add focused production-helper regressions for both reasoning and functionCalling.
  • Bring README.zh-CN.md to parity with the English fail-closed and acknowledgment contract.
  • Keep the committed snapshot as the single input authority; do not add a second shrink-state mechanism.

The separate question of hard-crash/concurrent atomicity across the three generated files is outside this issue; existing semantic consistency checks make that a recoverable hardening concern rather than the capability-loss defect fixed here.

中文

问题

在 PR #3565 的 exact head c658f3c15fcfc80fa0b16eb9467de471ee953d52 上,refresh:model-metadata 没有把布尔能力从 true 变为 false 视为 projection removal。

scripts/sync-model-metadata.mjs 会递归记录被删除的对象字段和数组成员,但 collectProjectionRemovals() 遇到 primitive 旧值时会直接返回。因此以下两种受支持的上游变化目前都能在不传 --accept-upstream-removals 的情况下直接发布:

  • reasoning: true -> false
  • tool_call: true -> false,投影后为 capabilities.functionCalling: true -> false

使用生产 helper 的定向复现确认:两次 refresh 都成功,并把 false 能力写入快照。现有 8 个 focused tests 全绿,是因为它们覆盖了模型、pricing 字段、对象 key 和数组成员的删除,没有覆盖布尔能力丢失。

这违反了文档中“已提交 capability 消失时 refresh 必须 fail closed”的契约;它可以在没有显式审查/确认的情况下,把模型移出支持工具调用的目录,或移除 reasoning controls。

验收条件

  • 将模型 capabilities projection 下的 true -> false 识别为 removal。
  • display name、status、limit、price 等普通标量修正仍可不带 removal flag 刷新。
  • 保留 --accept-upstream-removals 作为显式确认路径。
  • 为 reasoning 和 functionCalling 各补一个直接调用生产 helper 的回归测试。
  • 让 README.zh-CN.md 与英文版的 fail-closed / 显式确认契约保持一致。
  • 保持 committed snapshot 为唯一输入 authority,不增加第二套 shrink state。

三个生成文件在进程硬退出/并发执行时的跨文件原子性属于另一项可恢复的 hardening,不在本 issue 范围内。

Activity

  1. self-assigned this
    on Aug 23, 2026
  2. added a commit that references this issue on Aug 23, 2026
    3c2ace9
  3. added a commit that references this issue on Aug 23, 2026
    1ffed86
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions