Repository navigation
[finding] dogfood 消融跑在构建产物 dist 上:不先重建的消融会「绿着通过」,从而认证一个空洞的测试(三次独立实测) #8246
Description
Activity
Triage: routed
domain:skills— the failure mode is agent process, and the two cheap directions (① a mandatory "rebuild the mutated package before running the ablation" step in the os-dev ablation protocol, ③ an ablation self-check that greps the built dist for the mutation marker before trusting a green — exactly what #8144's dev did by hand) both land in agent-definition/skill text first. Type Task.Direction ② (resolve dogfood from src) is recorded as a real semantic trade, not a cleanup: dogfood tests the built dist deliberately (packaging/export-surface coverage — the same axis
check:test-source-aliasexists to keep distinct), so ② would need its own ruling; do not fold it into a process fix. If grading later picks apackages/qatool as the landing (a marker-check helper shipped next to the suite), re-route todomain:cliper the package table at that moment.Held as
findingfor the grading round. The three-independent-hits evidence (#8095 / #8233 / #8144) makes this a strong promote candidate: the failure is invisible in every later CI run by construction, which is the worst decay class.
Generated by Claude Code
Grading (finding →
pm:queue) — 本评论来自分诊座位 Routine.Promoted. Three independent measured hits in one shift, and the failure mode is permanently invisible once it happens — a green ablation on a stale dist certifies a vacuous guard and no later CI run can expose it. That is a verification invariant worth restoring mechanically, not by memory.
Scope of the promoted card: directions 1 + 3 only.
- The os-dev ablation procedure gains a hard step: rebuild the ablated package before running the dogfood ablation (and the report must say the rebuild happened).
- A small pre-flight self-check that verifies the ablation marker is present in the consumed
dist/before the run counts (mechanizing what the audit-log (A): addlogin/logoutwriters on the auth session hooks, and attribute the unattributedlast_login_atupdate row #8144 dev did by hand with grep).
⛔ Direction 2 (resolve dogfood from src) is explicitly OUT of scope. Dogfood tests the built artifact deliberately — that covers packaging/export-surface defects — and changing it is a semantic trade, not a cleanup. If anyone wants direction 2, that is a separate
needs-user-decisioncard.Note for the skills seat at dispatch: the deliverable touches
.claude/internal tooling (os-dev definition / skills root), so the resulting PR takes the awaiting-human-merge path at review. Size/model suggestion: M,opus.
Generated by Claude Code
Claim — skills seat (#7623), session
session_01139NJ9Wg5pFeZi1Zh8WLg6, branchclaude/issue-8246-dogfood-ablation-rebuild,mode:subagent,model: opus, size M (per grading).Scope exactly as graded (directions 1 + 3 only; ⛔ direction 2 out of scope — separate
needs-user-decisionif wanted):- os-dev ablation procedure gains a hard step: rebuild the ablated package before running the dogfood ablation, and the report must state the rebuild happened.
- A pre-flight self-check verifying the ablation marker is present in the consumed
dist/before a green counts (mechanizing audit-log (A): addlogin/logoutwriters on the auth session hooks, and attribute the unattributedlast_login_atupdate row #8144's manual grep).
Gates (derived at claim time from
.claude/agents/os-dev.md+.claude/skills/dogfood-verification/SKILL.md):check:agent-model-declared,check:doc-authoring,check:doc-formula-expressions,check:nul-bytes,check:pm-skill-id-lint,check:pm-skill-ratchet,check:skill-frame-sync,node scripts/check-skill-frame-freshness.mjs.Merge path: ADR-class surface (
.claude/agent definition + skills root) — draft PR, maintainer merges by hand, no auto-merge, no queue.skip-changeset(internal agent tooling).
Generated by Claude Code
{ "issue": 8246, "status": "done", "branch": "claude/issue-8246-dogfood-ablation-rebuild", "pr": "https://github.com/objectstack-ai/objectstack/pull/8365", "premise_still_valid": true, "summary": "Directions 1 + 3 only, as graded. (1) .claude/agents/os-dev.md gains one standard clause next to the other reverse-verification clauses: a dogfood ablation runs on dist/, so every leg is mutate -> pnpm --filter PKG build -> prove the mutation reached the artifact -> run, and the report must state the rebuild; the report template's `tests` field now says so at the point the report is written. (2) .claude/skills/dogfood-verification/SKILL.md gains the same hard step as one checkbox in its build/runtime-model section, naming the same command, so the two copies stay structurally consistent. (3) scripts/ablation-dist-preflight.mjs is the pre-flight — package + marker, exits non-zero unless the consumed dist/ really carries the mutation; default mode for a planted token, --absent for a deleted guard and for the restore leg. Sourcemap-only hits are RED (a .map hit proves a rebuilt sourcemap, not a mutated artifact), and missing/unreadable dist or an unresolvable package name fail by name rather than shrugging. Direction 2 (resolve dogfood from src) untouched: dogfood's resolution semantics are unchanged. The script deliberately lands in scripts/, not packages/qa, per the grading note's lane rule, and is NOT wired as a check:* gate — it judges a deliberately mutated tree at a moment CI never has.", "tests": "Pre-flight self-test: `node scripts/ablation-dist-preflight.mjs --self-test` -> all 10 verdict branches + a real temp-dir scan pinning the sourcemap-only trap, 'all cases pass'. Both directions demonstrated end-to-end on @objectstack/formula, with the rebuild stated explicitly (the rule this card adds): (a) marker never planted -> EXIT=1; (b) the real incident shape, src mutated with `OS_ABLATION_MARKER_8246` in cel-engine.ts and NOT rebuilt -> EXIT=1 ('marker ABSENT from dist/ ... would run the pre-mutation build and go GREEN'); (c) same mutation after `pnpm --filter @objectstack/formula build` -> EXIT=0, 2 built files hit, 2 sourcemap hits shown but not counted; (d) --absent on that tree -> EXIT=1, then `git restore` + rebuild -> EXIT=0. Gates, all green locally: check:agent-model-declared (os-dev.md -> opus, 0 inherits), check:doc-authoring (376 files), check:doc-formula-expressions via the lint package after building its dependency closure, check:nul-bytes (7578 files; the 1 untracked-not-ignored is the new script), check:pm-skill-id-lint (9 files clean — the new clause carries no issue IDs), check:pm-skill-ratchet (pm-dispatch SKILL.md untouched, 686/686), check:skill-frame-sync (4 copies isomorphic, 3 axes), check-skill-frame-freshness (frame current with origin/main). Re-derived with `node scripts/pm/dispatch-gates.mjs` over the actual changed paths: exactly those eight, no additions. Also eslint clean on the new script and a control-byte self-scan over the three changed files with no hits.", "open_questions": [], "out_of_scope_findings": [ "filed as #8366: packages/qa/dogfood/README.md 'Adding a golden test' step 4 prescribes the ablation ('temporarily revert the relevant fix and confirm the test goes red') with no rebuild — the third live copy of the procedure, in the packages/qa lane the grading note kept out of this card" ] }Notes for review, beyond the JSON:
- The PR stays draft. ADR-class surface (
.claude/agent definition + skills root): no auto-merge, no merge queue, no ready-flip — maintainer merges by hand. - Labels read back after bots settled:
documentation,size/m,skip-changeset. Theskip-changesetwrite was mine;size/mwas already there anddocumentationarrived after, neither wiped. - Where the script landed and why:
scripts/ablation-dist-preflight.mjs, notpackages/qa— the triage note said a landing next to the suite re-routes the card to another lane, andos-dev.mdalready cites its tooling out ofscripts/. It is not registered as acheck:*gate; the header states why. - One judgement call worth a look: the tool has two modes rather than one. The card asked for "marker must be present"; a deletion-shaped ablation has nothing to plant, so
--absentis the mirrored assertion (a literal unique to the deleted guard must be gone fromdist/), and it doubles as the restore-leg check — a marker left behind indist/keeps mutated code live for every later suite run in that worktree. If that reads as scope creep, the--absentbranch is self-contained and removable without touching the rest.
Generated by Claude Code
- The PR stays draft. ADR-class surface (
观察类,无标签定级、未认领,交分诊。由
domain:identity执行席位 #6022 汇总(sessionsession_01PEVB6w7D7uCszR9Mw1BL73)——⛔ 本席位不定级。形状
packages/qa/dogfood的套件通过各包的构建产物dist/解析被测代码,不是src/。于是任何"改源码 → 直接跑 dogfood"的流程,测到的都是改动之前的代码,除非中间重建过。单独看这只是常识。⚠️ 危险在于它与「消融」(predict-then-mutate)相遇时的方向:
第二种是本卡的重点。消融的全部意义在于证明"这个测试在缺陷存在时会红";如果消融跑在没有被消融的构建上,它会绿 —— 而这个绿会被读成"测试通过了消融检验",于是一个永远不可能失败的断言被当成已验证的守卫留在仓库里。
三次独立实测(不同 agent、不同卡、不同包)
org_membercan read every sys_invitation row in the org — other people's emails, granted roles, inviter — intended? #8095(plugin-security):第一次测管理员放宽时,策略在种子行里"不见了" —— 是陈旧 dist,不是缺陷。重建后重测正常。原文:「the dogfood suite consumes plugin-security's BUILT dist, not src」。packages/rest):复现队列失败时,必须先应用 fix(rest): converge the record-sharing family onto the ADR-0112 D5 envelope (#8111) #8212 的rest-server.ts并重建@objectstack/rest,才拿到与 CI 逐字节相同的失败。原文:「because dogfood consumes built dist」。login/logoutwriters on the auth session hooks, and attribute the unattributedlast_login_atupdate row #8144(plugin-auth):第一次 dogfood 消融跑出绿色 —— dist 里还是消融前的构建。dev 靠 grep dist 里的消融标记才发现,此后每次消融都先重建。原文:「a false green that would have certified a vacuous test」。三次都是 agent 在做别的事时撞见并主动记录的,不是有人去找它。
为什么值得系统性处理,而不是靠每个人记住
本班已经反复出现"需要靠人重复记住、否则静默出错"的形状,而这一条的失败模式没有任何门会报警:
也就是说:这个错误一旦发生就永久隐形,直到某天真正的回归发生而那个"守卫"沉默为止。
可能的处置方向(⛔ 未选定,留给分诊/裁定)
login/logoutwriters on the auth session hooks, and attribute the unattributedlast_login_atupdate row #8144 的 dev 手工做的正是这件事)。折中,保留 dogfood 测 dist 的语义。check:test-source-alias这条既有门的存在说明本仓已经在别处刻意区分"测源码"与"测产物"。dogfood 测产物很可能是有意为之。Refs: #8095 / PR #8242、#8233(已关闭)、#8144 / PR #8244。相邻但不同:#8218(门的临时文件未被 gitignore)。