Repository navigation
Stop 23 (Phase 8 — unattended agents, gh-aw): closed at n = 0, and the instrument built to catch a stale quote had five defects that manufactured them - #134
Merged
Conversation
Second ◇ stop of spine row 22–23, "extract only, as stop 7". §4 steps 1–2 plus the
extract. No experiment file, no registered prediction, no benchmark run, no overlay,
nothing spent on the agent under test.
Seven pages read fresh 2026-09-27 — the scaffold's five plus Threat detection and
Triggers, added because two exit-gate clauses cannot be answered from the five.
check-links.sh: ok=73 moved=11 blocked=2 unverified=0 broken=0, exit 0.
The headline is a failed check on this project's own workbook. The August 2026 extract's
boldest display quote — "The agent never receives write tokens directly." — is no longer
in the page it cites. evidence/p08/verify-quotes.sh (new) fetches all seven pages and
greps for the 29 sentences the workbook quotes: found=28 absent=1, exit 2, and the one
absence is ours. check-links.sh proves a URL resolves; nothing here proved a quotation
still appears in the resolved page. Its fixture set verify-quote-checker.sh passes 10 of
10 over all four registered exit codes, and case B was re-derived by hand.
Findings of record, each quoted verbatim in the workbook:
- The closed output vocabulary roughly doubled and now holds merge-pull-request,
approve-workflow-run, push-to-pull-request-branch, dispatch-workflow and third-party
writers. The mechanism is unchanged and the separation holds, but a closed vocabulary
bounds the attack surface exactly as tightly as its membership, and the membership is
the vendor's to change between two readings of the same page.
- The injection defence is L2, not L1, and its judge is a language model — "the same AI
engine as your main workflow". It fails closed on its own error, which is right, and the
opposite of check-links.sh degrading a curl 000 to a non-fatal unverified.
- Protected files is the only rule-based lane, and its default routes to a human rather
than refusing. Its threat model names CLAUDE.md and .claude/settings.json — this
project's own treatment-delivery channel.
- The spine's L1 label for this phase is right about the credential and wrong about
everything that inspects content. Qualified, not overwritten.
- roles: is an exact-match allowlist, not a privilege threshold. Third occurrence here of
a permission field whose direction must be read rather than assumed.
- stop-after: is this phase's L3 noise-kill rule shipped as L2, and the cheapest thing on
these pages to port.
- The vendor now ships hypothesis:, min_samples:, decision:{minimum_effect, confidence}
and PROMOTE/REJECT/INCONCLUSIVE/EXTEND. Two designs converged on four rows. But
hypothesis: is free text nothing parses — not a registration — and "chooses the
least-used variant" is balanced assignment, not randomisation.
- .md → .lock.yml: the reviewed file is not the executed file, and stale / out of date /
recompile / --verify are all absent from the page that teaches workflow creation. Same
gap as position 9's tools: list with a build step in the middle.
Labs 8.1–8.4 deferred with what each still owes; 8.3's owings changed most, because
"security detections fire" is two outcomes at two layers. lab#10 stays open on that
basis, comment posted, card moved to In Progress and read back.
Boundary 2 (§4 steps 11, 13, 13a, 14) is the next session's.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…act written status: running. Not a halt: blocked_on_author is empty and no §7 bullet is matched. prompt_sha a47590a1e61d re-computed and unchanged; all 22 validator files re-checked by name, none new. The new current-state block goes at the top, newest-first as this file has always been kept. The superseded stop-22 status KEY LINE is commented out so exactly one `^status:` is live — the driver greps `^status: *(done|blocked)` and two live keys is the one failure this file's own header warns about. Its text and its whole 1330-line comment block are kept verbatim. §0a re-run in full at the author's instruction, the tenth consecutive session: six ok, two FAIL, one RED-by-decision. Row 2 is the ninth consecutive ollama-cloud stall (903 bytes, 0 sections, re-derived by hand) so boundary 2's review goes to -P codex with the acceptance gate recorded DID NOT RUN. Row 6a's codex isolation leak reproduces for the third time and is deliberately not fixed — the fix changes the run environment every later stop inherits. Row 3b was reported TIMEOUT by the subagent and is ok: the sheet was already complete on disk, read by hand, cell-for-cell identical to eleven earlier sheets across 25 days. §4b's re-derive rule has now overturned a subagent verdict in five of six sessions. Row 6b: run 59ea41b2-8e84-479c-ac72-3623ad11c3de, the pinned model checked not assumed, all seven customization hashes null under ISOLATE_USER_SETTINGS=1 — and the unobservable half stays unobservable, because a paths(scalars) scan for /hook/i over the whole record returns [] and §0a asks for a hook-execution count that does not exist in the schema. Three author_notes items added, none a blocker: a repository-wide quote-staleness checker (this stop built a phase-scoped one and §6 forbids generalising it here); check-links.sh failing OPEN on a curl 000 where gh-aw's detector fails closed; and INCONCLUSIVE vs EXTEND as two decision-rule rows rather than one NOT DETECTABLE, which would change how three closed stops read and is therefore the author's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The branch: line promised the push result would be in the state file rather than only in a terminal. Recorded: new branch pushed and tracking origin, two commits on it. No PR — that is §4 step 14 at boundary 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e, exit gate answered §4 step 11 — the six keys of build/README.md#after-every-step, plus the two scaffold questions not covered by a key. The headline stays what boundary 1 found: a verbatim quote in this project's own August extract went stale in 49 days and nothing executed to catch it. §4 step 13 — the §5 validation table, one row per gate clause, with the layer of the PROOF and not of the artifact. Every command in the re-derivation column was re-run immediately before the section was written: ./evidence/p08/verify-quote-checker.sh 10 passed, 0 failed, exit 0 ./evidence/p08/verify-quotes.sh found=28 absent=1, exit 2 (the ABSENT line is ours) ./tools/check-links.sh ok=73 moved=11 blocked=2 unverified=0 broken=0, exit 0 n = 0 runs, and it is COUNTED rather than asserted: 0 of 740 runs on the API carry a stop-23 experiment key. Independence check: there are no arms to be independent of. Exit gate — four clauses answered, one answered QUALIFIED and that is the correction this stop owed its own scaffold: "read-only default" is false without a subject. The agent job is read-only; the workflow is not — with no safe-outputs: block at all, create-issue is automatically enabled with max: 1. And "was this the agent, or the harness?" is NEITHER. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s stop's own instrument
TWO INVOCATIONS, -P codex, -n 2 each, sequential. No acceptance verdict exists and it is
recorded DID NOT RUN, not UNDECIDED and not a pass: the gate is an opencode route and that
provider's weekly limit has stalled §0a row 2 for ten consecutive sessions. Both panels ran
clean (codex ok 62s/63s and 24s/25s), so the findings are results.
findings/opencode/review-README-20260927T201538Z.md 23136 B, 44 sections
findings/opencode/review-verify-quotes-20260927T202016Z.md 6555 B, 9 sections
THE INSTRUMENT WAS WRONG, IN FOUR PLACES, ALL THE SAME SHAPE: a failure to fetch reported as
documentation drift — the house failure mode aimed at the tool built to catch it.
* curl had no --fail: a 404 with a non-empty body exits 0, strips to real text, and every
quote on that page reports ABSENT at exit 2.
* an empty CACHED page was accepted where an empty live response or fixture exits 3. Same
content, two verdicts, decided only by where it came from.
* an undeclared page key read a missing file with stderr discarded and printed ABSENT, so a
one-character typo was indistinguishable from real drift.
* strip keeps page chrome. DISPUTED and registered as a scoped limitation in the header.
PROVED AGAINST THE PRE-FIX SCRIPT rather than asserted: cases J and K each return
`exit 2, found=20 absent=9` at git show HEAD:evidence/p08/verify-quotes.sh. ONE EMPTY CACHED
PAGE AND THE OLD CHECKER REPORTED NINE QUOTATIONS AS STALE.
After the fix: shellcheck exit 0 on both scripts, 14 of 14 fixtures over all four exit codes,
and the live run reproduces found=28 absent=1 exit 2 with the same absent sentence — so the
stop's headline is unchanged by the fix, which is the only reason it may still be stated. The
fail-closed path was OBSERVED, not argued: exit 3 on a page that timed out.
AND THE SAME LAYER MISTAKE AS STOP 22, AGAIN, CAUGHT BY THE SAME REVIEWER AND NOT BY ME:
`staged: true` was L1 and is L2 — the write request is still expressed and something executes
and skips it. So ONE subject row survives step 1, not two. Every count that quoted two is
corrected.
Also fixed from the review: `roles:` does not reach an actorless `schedule:` trigger, which
makes the surface LARGER than the gate's sentence implied; the vocabulary/perimeter claim is
scoped to the documented vocabulary and marked unmeasured for an already-compiled .lock.yml;
"29 network fetches per invocation" was wrong by four-fold (seven pages, 29 sentences);
fail-closed is right for a check whose output GATES A CLAIM, not universally; a redacted
`19:2xZ` identifies no run and now cites an artefact; and the Commit block is annotated as the
deferred labs' deliverables. TWO DISPUTED IN WRITING: the compile-step correspondence check
(the finding restates the section's conclusion as an omission) and the deliberately broad jq
pattern (a superset matching nothing proves the subset matches nothing). FIVE REGISTERED AS
DEBTS of the deferred labs, not fixed — §6 forbids designing a future step's decision rule.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, and found a fifth defect THE GATE THAT HAS NOT RUN IN TEN SESSIONS RAN ON THIS ROUTE — lab-acceptance · minimax-m3 — and BLOCKED. Both line-level runs raised the same thing independently: strip() had NO EXIT-STATUS CHECK. `set -uo pipefail` does not catch it (`-e` is absent and the status was never read), so a crashing python3, a missing module or ONE NON-UTF-8 BYTE left an empty .txt, the matcher read it, and EVERY QUOTE PRINTED ABSENT AT EXIT 2 — an internal processing failure reported as documentation drift, in the same format as a real finding. PROVED AGAINST THE PRE-FIX SCRIPT, NOT ASSERTED: case L (a python3 on PATH that exits 1) OLD exit 2, found=0 absent=29 case M (a page with a non-UTF-8 byte) OLD exit 2, found=0 absent=29 ONE BROKEN INTERPRETER AND THE OLD CHECKER DECLARED THE ENTIRE EXTRACT FABRICATED. The four round-1 defects each manufactured nine phantom findings; this one manufactures all 29. It is the sharpest instance of this project's house failure mode found so far — a control whose failure is indistinguishable from, and louder than, the finding it was built to make. Fixed: strip failure or empty output -> exit 3. shellcheck exit 0 on both scripts, 16 of 16 fixtures over all four registered exit codes, and the live run still reproduces found=28 absent=1 exit 2 with the same absent sentence — so the stop's headline is unchanged by FIVE fixes to the instrument that produced it, which is the only reason it may be stated. Round 3 not run and the loop closed at two: §4a caps at three, the blocking finding is fixed, and the gate is a different model from the line-level pass, so a third round would be a third detection threshold rather than an answer. Two rounds, five defects, all fixed, one finding (page chrome) disputed in writing and registered as a scoped limitation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous Position block is kept VERBATIM under `## Position — superseded 2026-09-28 at the stop-23 close`; nothing of it is deleted. Two headlines, and the second is the sharper one: a verbatim quote in this project's own August extract went stale in 49 days with nothing executing to catch it — and the instrument built to catch it had FIVE defects of a single shape, every one reporting a FAILURE TO FETCH OR PARSE as DOCUMENTATION DRIFT. One empty cached page manufactured nine phantom stale quotations; one broken interpreter or one non-UTF-8 byte manufactured all twenty-nine. Also: `staged: true` was L1 and is L2, the same layer mistake as stop 22 and caught by the same reviewer, so ONE subject row survives step 1 and not two; `roles:` does not reach an actorless `schedule:` trigger, which makes the surface larger than the exit gate's own sentence implied; and the vocabulary/perimeter claim is scoped to the documented vocabulary and marked unmeasured for an already-compiled .lock.yml. Board check is RED and EXPECTED — the boards are the author's by decision 12 item 4 and are NOT republished. Required marker digest, re-derived by RUNNING ./tools/check-board-freshness.sh after the last HANDOFF.md edit rather than by hand: c1e2b847fce9. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the ◇ extract-only stop at spine position 23 — Phase 8, unattended agents (
gh-aw) — atn = 0runs, which §3's itinerary row22-23registers a ◇ stop to produce and is not a shortfall. Counted rather than asserted: 0 of 740 runs on the API carry a stop-23 experiment key.Two results, and the second is the sharper one
1 — a verbatim quote in this project's own extract went stale in 49 days and nothing executed to catch it. The August 2026 extract of this same phase carries, as a bold display quote, "The agent never receives write tokens directly." That sentence is not on the page it cites. The page's equivalent today is "all without giving the agentic portion of the workflow any write permissions" — the claim survived the rewording, the quotation did not. Two other August quotes were re-checked and are still verbatim, so this is one sentence, not a rewritten page.
check-links.shproves a URL resolves; nothing proved a quotation still appears in the resolved page.2 — the instrument built to close that gap had five defects, all of one shape: a failure to fetch or parse, reported as documentation drift. Found across two §4a rounds, every one proved against the pre-fix script rather than asserted:
curlhad no--failABSENTat exit 2exit 2, found=20 absent=9— while an empty live response or fixture exits 3. Same content, two verdictsexit 2, found=20 absent=9— a one-character typo,2>/dev/nullswallowing the errorstriphad no failure guard (the acceptance gate's blocking finding)exit 2, found=0 absent=29— the entire extract declared fabricatedstripkeeps page chromeOne empty cached page manufactured nine phantom stale quotations; one broken interpreter or one non-UTF-8 byte manufactured all twenty-nine — at exit 2, in the same format as a real finding. That is this project's house failure mode aimed squarely at the tool built to catch it, and the same shape as the stop-8 contamination guard that would have excluded the treatment arm and reported a null.
After the fixes: ShellCheck exit 0 on both scripts, 16 of 16 fixtures over all four registered exit codes, and the live run still reproduces
found=28 absent=1, exit 2, same absent sentence — so the headline is unchanged by five fixes to the instrument that produced it, which is the only reason it may still be stated.3 — the same layer mistake as stop 22, again, caught by the same reviewer and not by me.
staged: truewas labelled L1; applied in order the write request is still expressed and something executes and skips it → L2. Exactly one of ten subject rows survives step 1 (the safe-output separation), not two.§4a review — two rounds, and no acceptance verdict at round 1
findings/opencode/review-README-20260927T201538Z.md— 23 136 B, 44 sections, the workbook.## Acceptancereads "The gate failed to run (opencode exit 1)" → recordedDID NOT RUN, notUNDECIDEDand not a pass.findings/opencode/review-verify-quotes-20260927T202016Z.md— 6 555 B, the tool, round 1.findings/opencode/review-verify-quotes-20260928T040312Z.md— the FINAL review file, round 2, and the gate RAN here (lab-acceptance · minimax-m3) and returnedREJECTon defect 4. Fixed infbf9d7d; round 3 not run, loop closed at two per §4a's cap.Both panels ran clean (
codex ok 62s/63s,24s/25s), so the findings are results, not stalls. One family only, so recurrence reads1/1; the four findings both runs raised are stated as 2 of 2 runs.Per finding: the sha that fixed it, or the dispute
staged: truelabelled L1f7b1e42roles:treated as reaching all thirteen triggersf7b1e42.lock.ymlf7b1e42f7b1e4219:2xZidentifies no executionf7b1e42f7b1e42f7b1e42f7b1e42f7b1e42jqpattern is broader than the stop's key setf7b1e42stripkeeps page chromefbf9d7dcurl --fail, empty cache, undeclared keyf7b1e42stripfailure guard (gate: REJECT)fbf9d7df7b1e42Not sent to the harness, and named here as §4a requires:
HANDOFF.mdandSOURCES.md(neither is a contract this stop authored), and nothing underfindings/,.claude/or.github/—rtkhides dotfile paths and the harness would review nothing and exit 0.§5 validation
The table is in the workbook, one row per gate clause, with the layer of the proof rather than of the artifact. Every command in its re-derivation column was re-run immediately before the section was written, and the output quoted is that run's:
Independence check: there are no arms to be independent of. One clause of the phase's own exit gate — "read-only default" — is answered qualified, because it is false without a subject: the agent job is read-only, the workflow is not (with no
safe-outputs:block,create-issueis still enabled atmax: 1). "Was this the agent, or the harness?" is neither.Expected red
The board check is RED by the author's standing decision 12 item 4 — the boards are the author's and are not republished here. Required marker digest, re-derived by running
./tools/check-board-freshness.shafter the finalHANDOFF.mdedit rather than by hand:c1e2b847fce9. Nine green and one red is the mergeable state, aslab#132,lab#130,lab#122andlab#117all were.Deferred
Labs 8.1–8.4 are all DEFERRED with their debts now written down, so
lab#10stays open — a Phase issue closes only when its gate is met from measurement.Validator files processed
All 22, newest
findings/track-b-validation-2026-09-08-4.md; none new since, re-checked one grep per basename.🤖 Generated with Claude Code