Repository navigation
Ablation edits that match ZERO and exit 0 silently invalidate the evidence reviews run on — twice in one afternoon #9914
Description
Activity
Claim: PM loop round 16
Session:session_01XqDQYVU5smx29ts9pAErja
Branch:claude/issue-9914-ablation-confirmation-contract
Worktree:objectstack-issue-9914
Domain:domain:devx
File surface:.claude/agents/os-dev.md(stop on breach; explain in the report)
Container & model: S, mode:subagent, model: opus (node scripts/pm/dispatch-gates.mjs --tier .claude/agents/os-dev.mdrun this round: "no path-derived mandate … This line is a FLOOR, never a clearance").⚠️ Clause ① does NOT apply: it is scoped to.claude/skills/pm-dispatch/**, and this lands in.claude/agents/. Content limbno— a reporting-discipline line changes no contract's accept/reject behaviour.
Clause-②: no
Serial constraints cleared: swept all open PRs for any touching.claude/agents/— none. My lane currently has zero open PRs; everything dispatched this session has landed.⛔ GOVERNED SURFACE —
.claude/**. The resulting PR is human-merge-only: I will not arm it, will not queue it, and will not flip it out of draft. It hangs visibly for the maintainer, which is the same path PR #9856 / #9860 / #9890 took successfully today.Maintainer ruling on this card (2026-08-19, 「接受你的所有建议」): Shape 1 — the contract line now, shape 2 (a
scripts/pm/helper) only on recurrence.
Generated by Claude Code
- added a commit that references this issue
on Aug 19, 2026 { "issue": 9914, "status": "done", "branch": "claude/issue-9914-ablation-confirmation-contract", "pr": "https://github.com/objectstack-ai/objectstack/pull/10003", "premise_still_valid": true, "summary": "Shape 1 only, as ruled: a reporting-discipline clause in `.claude/agents/os-dev.md`, no gate, no CI check, nothing merge-blocking. WHERE (ruling 5): extended the EXISTING ablation bullet under the section 「标准条款住在这里,不住在你的派发词里」 — the bullet now spans lines 203-227; no new heading was invented. FOUR TOUCHES: (1) topic sentence, lines 203-204, now reads 「变异腿与还原腿都要重建,并在报告里说明你建了、变异又是怎么在磁盘上确认的。」; (2) the per-leg pipeline, lines 214-217, gains a first step — 改动 → **证明它真落到了磁盘**(下段)→ build → 证明它到达了 dist/ → 才读运行结果; (3) the clause itself, lines 219-227, verbatim: 「⛔ **落盘那一步的证据永远不是编辑工具的退出码** —— `sed`、`perl -i`、`str.replace`、`re.sub` 零命中时照样 exit 0,于是未改动的文件配上健康输出,读作一次成功的消融(实测同一下午两起:一次 `str.replace` 锚点没中,自测照报 `51 case(s) passed` 而夹具根本不在;一次 `perl -0pi` 零命中,靠另跑的一步 `git diff --stat` 才逮到)。这一步无条件成立,没有 build/dist 的消融同样要做 —— 两起都落在那儿。观察要**锚定你打算改的那处文本**:注入文本与被删文本各 `grep -c` 一次;裸 `git diff --stat` 非空或字节数变化只证明*有*改动 —— 同轮的其它编辑会替它变绿,等长替换的字节差本就是零。确认没过 ⇒ **这次消融没跑**,读数作废:改锚点重来,并在报告里说明第一次是空操作 —— 悄悄重跑到有东西落地,是把同一个缺陷复制到上一层。」; (4) the report template's `tests` field, line 333, now ends `an ablation states its rebuild and how the mutation was confirmed on disk`. Net +10 lines; the ratchet reads 368 lines against ceiling 399 (headroom 41 -> 31). H2 — YES, one adjacent rule exists and it is why this was extended rather than duplicated: the same bullet at old lines 203-217 already mandated a confirmation, but of the BUILD OUTPUT (改动 → build → 证明它到达了 dist/, mechanized as scripts/ablation-dist-preflight.mjs), so it is conditioned on a dist/ resolution path and structurally cannot fire on the surface where both occurrences happened — both were edits to scripts/check-*.mjs, which is never built and has no dist/. Hence the explicit 「这一步无条件成立,没有 build/dist 的消融同样要做 —— 两起都落在那儿」. AGENTS.md carries nothing on this at all (grep -c: ablation 0, 消融 0, 反向验证 0, against a control of 7 for `stash`, so the zero is a real zero and not a broken scan) — os-dev.md was the only home and there was no competing half-rule. The neighbouring 反向验证 bullets (164-165, 179-183, 199-202) cover direction and restore points, not landing; left untouched. H3 — the four listed observations do NOT all cover both cases; two are weaker than the list implies. `grep -c` of the injected text and sha256 before/after catch both. `git diff --stat` non-empty catches occurrence 2 (it is what actually caught it) but is weak on occurrence 1: it is non-empty for ANY change, and that was a 470-line change across two files, so an unrelated in-flight edit turns the observation green while the fixtures are absent. Byte-count delta catches both of these two but is structurally blind to a same-length mutation — flipping `!==` to `===` or one character in a regex has a zero delta, which is the commonest ablation shape. Common defect of the two weak ones: they answer 'did anything change?', not 'did the thing I meant change?'. So the clause adopts the stronger form actually used today (anchor to the intended text; grep -c the injected AND the removed text) and names the two weak observations with their blind spots rather than dropping them — taking the ruling's own invitation to improve the list on measurement. H4 — YES, it needed saying and is said: 确认没过 ⇒ **这次消融没跑**,读数作废, plus the obligation to record that the first attempt was a no-op; re-anchoring and re-running quietly reproduces the defect one level up. GOVERNED SURFACE: draft PR, auto-merge NOT enabled, not flipped out of draft, no ready-flip attempted. Not touched: .claude/skills/pm-dispatch/** (clause-① territory), and shape 2 was not built.", "tests": "No package code changed, so no pnpm test/typecheck applies. Gate union re-derived on the FINAL commit ff1d3307c3 with `node scripts/pm/dispatch-gates.mjs` and NO path arguments (it read merge-base 79c46da90 itself; exit 0, 1 path in the change set). It named 8 families; all 8 run, all green, each line below the gate's own verdict line, exit codes captured before any pipe (`cmd > log 2>&1; EXIT=$?`): check:agent-model-declared EXIT=0 '✓ 1 agent definition(s) under .claude/agents/ all declare a model — os-dev.md → opus'; check:doc-authoring EXIT=0 '✓ doc authoring guard: 379 files clean'; check:doc-formula-expressions EXIT=0 '✓ self-test: 24 cases passed' + '22 record-scoped formula example(s) across 404 files / 1414 TS blocks judged clean' (script name echoed 4x in the log, so this was not a zero-match --filter run); check:nul-bytes EXIT=0 'OK (scanned 6337 text file(s) -- no raw ASCII control bytes)'; check:pm-governed-merges EXIT=0 'self-test: 77 assertions'; check:pm-skill-id-lint EXIT=0 '15 file(s) clean'; check:pm-skill-ratchet EXIT=0 '.claude/agents/os-dev.md is 368 lines (ceiling 399; headroom 31)'; check:skill-frame-sync EXIT=0 '4 copies of the decision frame structurally isomorphic across 3 files'. pnpm install and the `@objectstack/lint^...` dependency build (needed by check:doc-formula-expressions) both ran under the shared lock via scripts/pm/os-verify-lock.sh — VERDICT command-exit 0, held 47s and 237s, waited 0s each. NO ABLATION was run: the change is instruction prose with no executable behaviour to mutate, so a mutate/restore pair would have measured nothing — stated rather than fabricated. MUTATION CONFIRMED ON DISK, by the discipline this PR adds, independent of the edit tool's exit code: the patch script asserted each of the 4 anchors matched exactly 1 and would have exited refusing otherwise; then from outside the edit tool — sha256 b818ea51… -> 9318e302…, bytes 30394 -> 31494 (+1100), grep -c of each injected string = 1, grep -c of each removed string = 0. One removed-side probe was mis-specified on the first pass (it named text the edit never removed, returning 1); corrected rather than read as a failure, and recorded because it is the same class of no-op this card is about. Control-byte sweep over the edited file (grep -naP '[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f]'): no hits. Labels read back after writing the UNION (never a bare set): size/s + skip-changeset both present, draft still true. Worktree removed without --force; node_modules deleted first. CI not waited on, per the 2026-08-10 ruling.", "open_questions": [], "out_of_scope_findings": [] }Two observations for the PM, neither filed as an issue (neither is a repo defect):
pnpm install --workspace-concurrency=2is rejected outright by pnpm 10.31 (ERROR Unknown option: 'workspace-concurrency'). The contract's toolchain trap is aboutpnpm --filter … run, where the flag is real and ordering matters — it makes no claim aboutinstall, so this is a misreading on my side, not a defect. Recorded in case the trap line is ever generalized.issue_read/get_labelscannot resolve a PR number (Could not resolve to an Issue with the number of 10003); the label read-back had to go throughsearch_pull_requests. MCP-server behaviour, not this repo.
Generated by Claude Code
Generated by Claude Code
- added a commit that references this issue
on Aug 23, 2026
Filed by the
domain:devxPM seat. Two independent occurrences in a single afternoon, in two different tools, by two different dev agents — and both were caught by luck rather than by process.The failure
An agent ablates a file to prove a gate fires. The edit command matches nothing and exits 0. The agent runs the gate, sees healthy output, and concludes the ablation proved something — when it ran against an unmodified file.
Occurrence 1 — PR #9875 (#9747)
Caught only by a later flip-test. Inside the very change that exists to make such operations announce themselves.
Occurrence 2 — PR #9815 (#9767)
Why this outranks an ordinary tooling bug
Ablation is the evidence this seat accepts work on. Reviews across this lane routinely turn on "same tree, two states, opposite verdicts" — that pair is what separates "I added a gate" from "I added a gate that catches the thing it was written for."
⇒ A silently-missed ablation does not produce a wrong answer. It produces a confident right-looking answer with no evidence under it, and the review cannot tell the difference. Every ablation claim in every report is only as good as the confirmation that the edit landed.
Both agents caught it. Neither caught it because of a rule — one hit it via an unrelated flip-test, the other because it happened to run
git diff --stat. The base rate of it being missed is therefore unknown and not zero.The rule the second agent derived, which is the fix
sed,perl -i,str.replace, andpython re.suball report success on zero matches. So the proof that an ablation landed cannot come from the edit command. It has to come from a different observation —git diff --statnon-empty, a byte-count delta, agrep -cof the injected text, or a sha256 before/after.Prior art in this repo, same shape one layer out: #9878 (a shallow clone answers a
--sincequestion plausibly and wrongly, with no warning) and #4690 ("zero is a broken scan, not a clean repo"). This card is that principle turned on our own verification method rather than on the gates.Candidate shapes (pick one; none is obviously right)
os-devdispatch contract: an ablation report must state how the mutation was confirmed on disk, by an observation independent of the edit tool. Cheapest, and it makes the omission visible at review time rather than preventing it..claude/**is a governed surface — human-merge-only, and per SKILL clause ① any.claude/skills/pm-dispatch/**edit is fable-mandatory.scripts/pm/helper that applies a mutation and refuses (non-zero, empty stdout) when the match count is not what the caller asserted. Not governed, mechanically enforced, and it composes with the existing worktree discipline. Costs adoption — a helper nobody runs fixes nothing (cf. scripts/pm/ci-failure.mjs's --self-test runs nowhere in CI, unlike its four sibling PM tools #9898).My read: 1, then 2 if it recurs. The failure is one of reporting discipline, and the two agents who hit it both had the instinct to check — what they lacked was a rule making the check mandatory and its absence visible. A helper (2) is stronger but only helps agents who choose it, and this failure happens precisely when an agent thinks it does not need help.
Refs: PR #9875 / #9747 (occurrence 1) · PR #9815 / #9767 (occurrence 2, and the rule) · #9878 · #4690 · #9898 (a self-test nothing runs).
Generated by Claude Code