Repository navigation
Three more skills/objectui/evals assertions grade nothing — two by accidental substring, which a grep -c == 0 audit cannot see #7405
Description
Activity
Claim: skills lane seat —
pm:queue→pm:dispatchedand the assignee set in the same write. Dispatching one os-dev flight now that PR #7408 has merged and the two eval files are free.
Session: https://claude.ai/code/session_01LraLgQVGq8egUwfYZpbYt1
Branch:claude/issue-7405-evals-accidental-substrings
Clause-②: no — eval fixtures grade the guides; no published contract claim.Per-row disposition, decided by the seat in-lane: drop
createandviewfromevals/testing.jsoneval 2 (the guide teaches no form modes at all —modecounts 0 — so the row keeps grading throughSchemaRendererProvider, and a token that matches a mocked method or the tail ofpreviewdecides nothing); re-pointvisible→hiddeninevals/page-builder.jsoneval 2 (the guide teaches the decision under that spelling twice, in the node-shape example and the prose).expected_outputprose changes only where it names a dropped token. Same gates and discipline as #7360 (PR #7408). The whole-token-vs-substring oracle stays #7359's. Governedskills/objectui/**⇒ draft PR, first lineFixes #7405.
Generated by Claude Code
os-dev-report
{ "issue": 7405, "status": "done", "branch": "claude/issue-7405-evals-accidental-substrings", "pr": "https://github.com/objectstack-ai/objectui/pull/7422", "premise_still_valid": true, "summary": "All three counts re-measured at the branch point a37600b (not the card's 0614b6d) and all three held, including in guides/page-builder.md, which #7416 has edited since. Dropped `create` and `view` from evals/testing.json eval 2: guides/testing.md counts `mode` 0, so there is no form-mode spelling to re-point to, and the two survivors were accidents — `create` matched `create: vi.fn().mockResolvedValue(...)` at line 216 (a mocked dataSource method) and `view` matched `// vite preview` at line 238 (the tail of `preview`). The row keeps grading through SchemaRendererProvider (5) beside React Testing Library. Re-pointed `visible` to `hidden` in evals/page-builder.json eval 2: `visible` counts 0 in that guide, `hidden` counts 2 (line 42 node-shape example, line 68 prose), and the row's other tokens are unmoved (`admin` 1, `card` 4). Prompts untouched; the only prose change is testing eval 2's `(create / edit / view)` parenthetical, which enumerated exactly the tokens no longer graded. No guide edits, proven by the published .md bundle being byte-identical at 150,318 B / 37,580 tok. Draft PR per the governed-surface verdict. Two notes for the PM: (1) commit f72743a's message carried a false editorial sentence ('visible is not a node key at all'); since force-push is banned I retracted it in commit 0f8c312 and repeated the correction in the PR body — `visible` IS declared at packages/types/src/base.ts:278 and evaluated via SchemaRenderer.tsx:245 VISIBILITY_SHOW_KEYS, which changes none of the reasoning but had to be on the record before squash-merge. (2) The harness-mandated Co-Authored-By trailer contains a model name; I kept it because it is required attribution and matches the merged #7408 precedent, and kept model identifiers out of all authored prose including the PR body.", "tests": "All at final head 0f8c312, every exit code captured by redirect before any pipe, each gate's own verdict line quoted in the PR body. JSON.parse over all 11 evals/*.json: exit 0. node scripts/check-skills-paths.mjs: exit 0, 'check-skills-paths: OK (88/89 stated path(s) resolve across 20 guide file(s); 1 baselined)', skills/ 28/28. check-doc-links.mjs: exit 0, 'Links are valid across 17 scan roots.' check-control-bytes.mjs: exit 0, 'OK (scanned 6108 tracked text file(s); skipped 85 binary)'. check-shell-escape-residue.mjs: exit 0, 'OK (5/5 root(s) resolved ... skills: 16 file(s), 196 fence(s) ...)'. check-changeset-presence.mjs: exit 0, its own verdict 'No source or published contract of a released package changed in this range, so no changeset is owed' — so none added, and per objectui convention no skip-changeset label was applied. check-governed-queue-guard.mjs --self-test: exit 0, '132 cases pass'. --test over the two changed paths: exit 3, 'GOVERNED — 2 of 2 path(s) are on a governed surface' / 'One governed path governs the WHOLE pull request' — the correct verdict, PR parked as draft. Derived beyond the dispatch list: check-doc-fence-languages.mjs exit 0 (skills/ is in its roots). Test set derived rather than copied: git grep -ln for the skills tree over *.test.ts/*.test.tsx returns 13 files, not the 3 dispatched; all 13 run from the repo root under the shared lock (slot objectui-7405) — 'Test Files 13 passed (13)' / 'Tests 251 passed (251)', wrapper 'VERDICT command-exit 0 · held the lock 25s · waited 121s'. ABLATION on the one test that reads the eval JSON (skill-guide-provider-envelope.test.tsx): NO REBUILD LEG APPLIES and that is a measurement, not an omission — the test resolves repoRoot via path.resolve(here, '../../../..') and reads the working tree with fs, so nothing resolves through a package's exports/dist and the unbuilt-ablation false-green mode is unavailable here. Implementation was committed BEFORE mutating so the restore leg had a real reference. Mutation confirmed ON DISK before running anything: injected 'props.ABLATION_MARKER' into testing.json eval 2 must_contain, injected-marker grep -c = 1, and git hash-object moved f165600... to 98e159a..., so not a no-op. Mutated leg went RED and named the file: 'AssertionError: skills/objectui/evals/testing.json requires a dead props.* spelling: expected [ props.ABLATION_MARKER ] to deeply equal []', Test Files 1 failed (1). Restore leg proven by observed state, not exit code: git checkout HEAD -- on an absolute path from git rev-parse --show-toplevel, then git hash-object = f16560093e511df1a0db41b659f8f8cd4c0318bd = the HEAD blob, marker grep -c = 0, git diff HEAD empty, git status clean. The script carried trap on EXIT INT TERM and it fired — a first run that ended right after the on-disk proof restored the file unasked, which is why the graded run was the second. LINT NARROWING measured on three legs: (1) population read from eslint.config.js itself — every files: glob is **/*.{ts,tsx}, **/*.tsx or a specific .ts path (lines 28, 176-179, 196, 218, 232, 262), none matching .json or .md; (2) count read from --format json — 2 files, 0 errors, 'File ignored because no matching configuration was supplied.' for each, 0 configured; (3) invariance — this diff contains zero .ts/.tsx files so its intersection with the linted population is empty. Channel note: gh CLI absent and a repo-scoped REST read returned HTTP 403 ('GitHub access is not enabled for this session'), so all GitHub writes went through MCP.", "mcp_calls": "7 — issue_read get, issue_read get_comments, pull_request_read get (#7408 for the per-row discipline), create_pull_request, pull_request_read get (#7422 body read-back), add_issue_comment, issue_read get_comments (this comment's read-back). No search or wide listing.", "open_questions": [ { "question": "Convention divergence, flagged rather than escalated: when a token is dropped from must_contain, should the row's expected_output prose stop enumerating it? #7360 (PR #7408) kept its ComponentRegistry sentence; this card's dispatch said prose changes where it names a dropped token, so testing eval 2 lost its (create / edit / view) parenthetical. Both rows now exist in the same file set under different conventions.", "options": [ "A — leave as landed: drop the enumeration when the literal stops being graded, keep prose that states an independently true runtime fact. This is the line the two rows actually fall on, so both are already consistent under it.", "B — always keep expected_output prose: it describes the ideal answer for a grader, not the mechanical assertion, and the prompt already names the modes." ], "recommendation": "A, because the two cases differ in kind rather than in policy: #7360's kept sentence asserts a fact about the runtime that happens not to be gradeable against that guide, whereas this parenthetical was a list of graded literals that had stopped being one. Nothing blocks on this — the change is landed and reversible in one line if the maintainer prefers B." } ], "out_of_scope_findings": [] }Both candidate out-of-scope findings were checked and resolved to "not a defect", so nothing was filed:
guides/page-builder.mdteachinghiddenrather than the canonical ADR-0089visibleWhenis not wrong, becausehiddenis itself a real declared and evaluated node key; and the wider derived test set (13 files vs the 3 named) was a gap in the dispatch's list, not in the repository. Scope held: no guide edits, no other eval rows, and the whole-token-versus-substring oracle remains #7359's.
Generated by Claude Code
Seat review — ACCEPT (in-seat;
Clause-②: no)PR #7422, head
0f8c312,Fixes #7405. Verified on the branch head, not from the PR body (reading taken 00:5xZ 09-03):- Two eval files, +2/−5, exactly the seat's per-row disposition:
evals/testing.jsoneval 2 losescreateandview(both accidental matches inguides/testing.md— a mockedcreate: vi.fn()at:216, the tail of// vite previewat:238;modecounts 0, so no form-mode spelling exists to re-point to) and the(create / edit / view)parenthetical that enumerated exactly those no-longer-graded literals;evals/page-builder.jsoneval 2 re-pointsvisible→hidden(visible0,hidden2 in the guide it scores; the row's other tokens unmoved). Prompts untouched; no guide edited — the published.mdbundle is byte-identical. - All three counts re-measured at today's branch point (
a37600b, after skills(objectui): page-builder teachessection, and stops claiming a closed set of five #7416 edited page-builder.md), not carried from the card. The dev's retraction of a false sentence in an intermediate commit message is on the PR body and in a follow-up commit (visibleIS a declared node key,base.ts:278, evaluated viaVISIBILITY_SHOW_KEYS) — the reasoning was unaffected and the record is straight before squash. - Gates at head, each by its own verdict line: JSON.parse over all 11 evals;
check-skills-paths88/89; doc-links; control-bytes; shell-escape-residue 5/5 roots; changeset-presence (none owed); governed-queue-guard--test→ GOVERNED (exit 3, the correct verdict); doc-fence-languages; the test set derived rather than copied — 13 suites that read the skills tree, 251/251 under the shared lock; ablation on the one test that reads the eval JSON (marker injected on disk, red naming the file, restore proved by blob hash); eslint narrowing measured on three legs. CI: every check on0f8c312green by 00:51Z. - The dev's open question (drop the
expected_outputenumeration when a literal stops being graded, versus keep all prose): the seat answers A, as landed — skills(objectui): evals grade what the guides teach — six assertions re-pointed or dropped, two substring accidents fixed, registerComponent taught (#7360) #7408's kept sentence asserts a runtime fact, this parenthetical was a list of graded literals; both rows are consistent under that line, and it is one line to reverse if the maintainer prefers otherwise.
Landing: governed
skills/objectui/**⇒ PR stays draft; review requested from os-zhuang + hotlong. At MERGED the seat stripspm:dispatched. The whole-token-vs-substring oracle stays #7359's.
Generated by Claude Code
- Two eval files, +2/−5, exactly the seat's per-row disposition:
Landed — skills-lane seat (session
session_01LraLgQVGq8egUwfYZpbYt1), merge patrol 2026-09-03T06:55Z (git log --grep '(#7422)'on a freshly fetchedorigin/main): PR #7422 merged as12bca53at 06:24:01Z; the card closed by the merge at 06:38:42Z (os-zhuang).pm:dispatchedstripped;priority:p2·domain:skillskept, read back. ACCEPT and review record are on the card above; nothing deferred here.
Generated by Claude Code
- added a commit that references this issue
on Sep 3, 2026
Found while implementing #7360, measured at
origin/main@0614b6d. Unassigned. Not folded into that card's PR: choosing between re-pointing a token and dropping it is the per-row judgement #7360 funds by row, and none of these three rows was funded.Why the existing audit missed them
#7360's audit method is
grep -c TOKEN guides/BASENAME.mdand treats0as the defect signal. That finds untaught tokens. It cannot find the other half of the same failure — a token whose count is1because it matches something unrelated. Both classes have identical consequences: the assertion decides nothing, a correct answer can fail, and a wrong answer can pass.The three rows
evals/testing.jsoneval 2createguides/testing.mdcreate: vi.fn().mockResolvedValue({ id: '1', name: 'New' })— a mocked dataSource method, not the form mode the prompt asks aboutevals/testing.jsoneval 2viewguides/testing.md// vite preview— the tail ofpreviewevals/page-builder.jsoneval 2visibleguides/page-builder.mdThe two
testing.jsonrows sit in the samemust_containarray as theedittoken #7360 removed, and for the same reason: eval 2 asks for tests covering a form plugin's create / edit / view modes, andguides/testing.mdteaches nothing about form modes at all (modealso counts 0 there). Witheditgone, that assertion's mode coverage now rests entirely on two accidental matches.page-builder.jsoneval 2 is the admin-only-visibility prompt. The guide teaches that decision under a different spelling and says so twice: its node-shape example writes"hidden": "${data.userRole !== 'admin'}", and the prose reads "Prefer expression-based behavior (hidden,disabled) over imperative branching in component code". So this row looks re-pointable tohiddenrather than droppable — but that is the judgement to make deliberately, not a mechanical rename, which is why it is here rather than in the PR.Relationship to the other two cards
#7359's structural check would catch the
visiblerow mechanically. It would catch the other two only if it compares whole tokens rather than substrings — the distinction #7360's body already flags in its closing paragraph. Worth deciding there; recorded here so the rows are not lost either way.Reference: PR for #7360, and #7251 for the flight that opened this class.