Skip to content

Stop 23 (Phase 8 — unattended agents, gh-aw): closed at n = 0, and the instrument built to catch a stale quote had five defects that manufactured them - #134

Merged
hermanngeorge15 merged 7 commits into
mainfrom
stop23/phase-8-agentic-workflows
Sep 28, 2026
Merged

hermanngeorge15 merged 7 commits into
mainfrom
stop23/phase-8-agentic-workflows

Conversation

@hermanngeorge15

Copy link
Copy Markdown
Contributor

Closes the ◇ extract-only stop at spine position 23 — Phase 8, unattended agents (gh-aw) — at n = 0 runs, which §3's itinerary row 22-23 registers a ◇ stop to produce and is not a shortfall. Counted rather than asserted: 0 of 740 runs on the API carry a stop-23 experiment key.

Two results, and the second is the sharper one

1 — a verbatim quote in this project's own extract went stale in 49 days and nothing executed to catch it. The August 2026 extract of this same phase carries, as a bold display quote, "The agent never receives write tokens directly." That sentence is not on the page it cites. The page's equivalent today is "all without giving the agentic portion of the workflow any write permissions" — the claim survived the rewording, the quotation did not. Two other August quotes were re-checked and are still verbatim, so this is one sentence, not a rewritten page. check-links.sh proves a URL resolves; nothing proved a quotation still appears in the resolved page.

2 — the instrument built to close that gap had five defects, all of one shape: a failure to fetch or parse, reported as documentation drift. Found across two §4a rounds, every one proved against the pre-fix script rather than asserted:

# Defect Pre-fix behaviour Fixture
1 curl had no --fail a 404 serving a non-empty body exits 0, strips to real text, every quote on that page prints ABSENT at exit 2 I
2 empty cached page accepted exit 2, found=20 absent=9 — while an empty live response or fixture exits 3. Same content, two verdicts J
3 undeclared page key exit 2, found=20 absent=9 — a one-character typo, 2>/dev/null swallowing the error K
4 strip had no failure guard (the acceptance gate's blocking finding) exit 2, found=0 absent=29 — the entire extract declared fabricated L, M
5 strip keeps page chrome DISPUTED in writing and registered as a scoped limitation in the script header —

One empty cached page manufactured nine phantom stale quotations; one broken interpreter or one non-UTF-8 byte manufactured all twenty-nine — at exit 2, in the same format as a real finding. That is this project's house failure mode aimed squarely at the tool built to catch it, and the same shape as the stop-8 contamination guard that would have excluded the treatment arm and reported a null.

After the fixes: ShellCheck exit 0 on both scripts, 16 of 16 fixtures over all four registered exit codes, and the live run still reproduces found=28 absent=1, exit 2, same absent sentence — so the headline is unchanged by five fixes to the instrument that produced it, which is the only reason it may still be stated.

3 — the same layer mistake as stop 22, again, caught by the same reviewer and not by me. staged: true was labelled L1; applied in order the write request is still expressed and something executes and skips it → L2. Exactly one of ten subject rows survives step 1 (the safe-output separation), not two.

§4a review — two rounds, and no acceptance verdict at round 1

  • findings/opencode/review-README-20260927T201538Z.md — 23 136 B, 44 sections, the workbook. ## Acceptance reads "The gate failed to run (opencode exit 1)" → recorded DID NOT RUN, not UNDECIDED and not a pass.
  • findings/opencode/review-verify-quotes-20260927T202016Z.md — 6 555 B, the tool, round 1.
  • findings/opencode/review-verify-quotes-20260928T040312Z.md — the FINAL review file, round 2, and the gate RAN here (lab-acceptance · minimax-m3) and returned REJECT on defect 4. Fixed in fbf9d7d; round 3 not run, loop closed at two per §4a's cap.

Both panels ran clean (codex ok 62s/63s, 24s/25s), so the findings are results, not stalls. One family only, so recurrence reads 1/1; the four findings both runs raised are stated as 2 of 2 runs.

Per finding: the sha that fixed it, or the dispute

Finding Disposition Sha
staged: true labelled L1 fixed — relabelled L2, every dependent count corrected f7b1e42
roles: treated as reaching all thirteen triggers fixed — scoped to actor-originated triggers; the scheduled path is not role-gated at all f7b1e42
vocabulary → "L3 perimeter" claimed more than was read fixed — scoped to the documented vocabulary, unmeasured for an already-compiled .lock.yml f7b1e42
"29 network fetches per invocation" fixed — seven fetches (one per page), 29 sentences searched locally f7b1e42
redacted 19:2xZ identifies no execution fixed — cites the artefact instead f7b1e42
"gh-aw's is the right one" (fail-closed) fixed — narrowed to a check whose result is quoted as evidence f7b1e42
Commit block lists files this stop did not create fixed — annotated as the deferred labs' deliverables f7b1e42
least-used variant tie-break fixed — scoped; only the confound's existence follows, not its shape f7b1e42
compile-step correspondence check disputed — the finding restates the section's own conclusion as an omission, and building the check would be an instrument for a workflow this project does not run (§6) f7b1e42
the jq pattern is broader than the stop's key set disputed — a superset matching nothing proves the subset matches nothing; the looseness is deliberate f7b1e42
strip keeps page chrome disputed — registered as a scoped limitation; body extraction needs a per-site selector that breaks on the next redesign, and these are full sentences fbf9d7d
curl --fail, empty cache, undeclared key fixed + fixtures I/J/K proving refusal f7b1e42
strip failure guard (gate: REJECT) fixed + fixtures L/M proving refusal fbf9d7d
five deferred-lab underspecifications registered as debts, not fixed — designing a deferred lab's decision rule is a future step's artifact (§6) f7b1e42

Not sent to the harness, and named here as §4a requires: HANDOFF.md and SOURCES.md (neither is a contract this stop authored), and nothing under findings/, .claude/ or .github/ — rtk hides dotfile paths and the harness would review nothing and exit 0.

§5 validation

The table is in the workbook, one row per gate clause, with the layer of the proof rather than of the artifact. Every command in its re-derivation column was re-run immediately before the section was written, and the output quoted is that run's:

./evidence/p08/verify-quote-checker.sh    16 passed, 0 failed          exit 0
./evidence/p08/verify-quotes.sh           found=28 absent=1            exit 2
./tools/check-links.sh                    ok=73 moved=11 blocked=2 unverified=0 broken=0   exit 0
shellcheck evidence/p08/verify-quotes.sh evidence/p08/verify-quote-checker.sh              exit 0

Independence check: there are no arms to be independent of. One clause of the phase's own exit gate — "read-only default" — is answered qualified, because it is false without a subject: the agent job is read-only, the workflow is not (with no safe-outputs: block, create-issue is still enabled at max: 1). "Was this the agent, or the harness?" is neither.

Expected red

The board check is RED by the author's standing decision 12 item 4 — the boards are the author's and are not republished here. Required marker digest, re-derived by running ./tools/check-board-freshness.sh after the final HANDOFF.md edit rather than by hand: c1e2b847fce9. Nine green and one red is the mergeable state, as lab#132, lab#130, lab#122 and lab#117 all were.

Deferred

Labs 8.1–8.4 are all DEFERRED with their debts now written down, so lab#10 stays open — a Phase issue closes only when its gate is met from measurement.

Validator files processed

All 22, newest findings/track-b-validation-2026-09-08-4.md; none new since, re-checked one grep per basename.

🤖 Generated with Claude Code

hermanngeorge15 and others added 7 commits September 27, 2026 21:46
Second ◇ stop of spine row 22–23, "extract only, as stop 7". §4 steps 1–2 plus the
extract. No experiment file, no registered prediction, no benchmark run, no overlay,
nothing spent on the agent under test.

Seven pages read fresh 2026-09-27 — the scaffold's five plus Threat detection and
Triggers, added because two exit-gate clauses cannot be answered from the five.
check-links.sh: ok=73 moved=11 blocked=2 unverified=0 broken=0, exit 0.

The headline is a failed check on this project's own workbook. The August 2026 extract's
boldest display quote — "The agent never receives write tokens directly." — is no longer
in the page it cites. evidence/p08/verify-quotes.sh (new) fetches all seven pages and
greps for the 29 sentences the workbook quotes: found=28 absent=1, exit 2, and the one
absence is ours. check-links.sh proves a URL resolves; nothing here proved a quotation
still appears in the resolved page. Its fixture set verify-quote-checker.sh passes 10 of
10 over all four registered exit codes, and case B was re-derived by hand.

Findings of record, each quoted verbatim in the workbook:

- The closed output vocabulary roughly doubled and now holds merge-pull-request,
  approve-workflow-run, push-to-pull-request-branch, dispatch-workflow and third-party
  writers. The mechanism is unchanged and the separation holds, but a closed vocabulary
  bounds the attack surface exactly as tightly as its membership, and the membership is
  the vendor's to change between two readings of the same page.
- The injection defence is L2, not L1, and its judge is a language model — "the same AI
  engine as your main workflow". It fails closed on its own error, which is right, and the
  opposite of check-links.sh degrading a curl 000 to a non-fatal unverified.
- Protected files is the only rule-based lane, and its default routes to a human rather
  than refusing. Its threat model names CLAUDE.md and .claude/settings.json — this
  project's own treatment-delivery channel.
- The spine's L1 label for this phase is right about the credential and wrong about
  everything that inspects content. Qualified, not overwritten.
- roles: is an exact-match allowlist, not a privilege threshold. Third occurrence here of
  a permission field whose direction must be read rather than assumed.
- stop-after: is this phase's L3 noise-kill rule shipped as L2, and the cheapest thing on
  these pages to port.
- The vendor now ships hypothesis:, min_samples:, decision:{minimum_effect, confidence}
  and PROMOTE/REJECT/INCONCLUSIVE/EXTEND. Two designs converged on four rows. But
  hypothesis: is free text nothing parses — not a registration — and "chooses the
  least-used variant" is balanced assignment, not randomisation.
- .md → .lock.yml: the reviewed file is not the executed file, and stale / out of date /
  recompile / --verify are all absent from the page that teaches workflow creation. Same
  gap as position 9's tools: list with a build step in the middle.

Labs 8.1–8.4 deferred with what each still owes; 8.3's owings changed most, because
"security detections fire" is two outcomes at two layers. lab#10 stays open on that
basis, comment posted, card moved to In Progress and read back.

Boundary 2 (§4 steps 11, 13, 13a, 14) is the next session's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…act written

status: running. Not a halt: blocked_on_author is empty and no §7 bullet is matched.
prompt_sha a47590a1e61d re-computed and unchanged; all 22 validator files re-checked by
name, none new.

The new current-state block goes at the top, newest-first as this file has always been
kept. The superseded stop-22 status KEY LINE is commented out so exactly one `^status:`
is live — the driver greps `^status: *(done|blocked)` and two live keys is the one failure
this file's own header warns about. Its text and its whole 1330-line comment block are
kept verbatim.

§0a re-run in full at the author's instruction, the tenth consecutive session: six ok, two
FAIL, one RED-by-decision. Row 2 is the ninth consecutive ollama-cloud stall (903 bytes, 0
sections, re-derived by hand) so boundary 2's review goes to -P codex with the acceptance
gate recorded DID NOT RUN. Row 6a's codex isolation leak reproduces for the third time and
is deliberately not fixed — the fix changes the run environment every later stop inherits.
Row 3b was reported TIMEOUT by the subagent and is ok: the sheet was already complete on
disk, read by hand, cell-for-cell identical to eleven earlier sheets across 25 days. §4b's
re-derive rule has now overturned a subagent verdict in five of six sessions.

Row 6b: run 59ea41b2-8e84-479c-ac72-3623ad11c3de, the pinned model checked not assumed,
all seven customization hashes null under ISOLATE_USER_SETTINGS=1 — and the unobservable
half stays unobservable, because a paths(scalars) scan for /hook/i over the whole record
returns [] and §0a asks for a hook-execution count that does not exist in the schema.

Three author_notes items added, none a blocker: a repository-wide quote-staleness checker
(this stop built a phase-scoped one and §6 forbids generalising it here); check-links.sh
failing OPEN on a curl 000 where gh-aw's detector fails closed; and INCONCLUSIVE vs EXTEND
as two decision-rule rows rather than one NOT DETECTABLE, which would change how three
closed stops read and is therefore the author's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The branch: line promised the push result would be in the state file rather than only in a
terminal. Recorded: new branch pushed and tracking origin, two commits on it. No PR — that
is §4 step 14 at boundary 2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e, exit gate answered

§4 step 11 — the six keys of build/README.md#after-every-step, plus the two scaffold
questions not covered by a key. The headline stays what boundary 1 found: a verbatim quote
in this project's own August extract went stale in 49 days and nothing executed to catch it.

§4 step 13 — the §5 validation table, one row per gate clause, with the layer of the PROOF
and not of the artifact. Every command in the re-derivation column was re-run immediately
before the section was written:
  ./evidence/p08/verify-quote-checker.sh   10 passed, 0 failed, exit 0
  ./evidence/p08/verify-quotes.sh          found=28 absent=1, exit 2 (the ABSENT line is ours)
  ./tools/check-links.sh                   ok=73 moved=11 blocked=2 unverified=0 broken=0, exit 0

n = 0 runs, and it is COUNTED rather than asserted: 0 of 740 runs on the API carry a
stop-23 experiment key. Independence check: there are no arms to be independent of.

Exit gate — four clauses answered, one answered QUALIFIED and that is the correction this
stop owed its own scaffold: "read-only default" is false without a subject. The agent job is
read-only; the workflow is not — with no safe-outputs: block at all, create-issue is
automatically enabled with max: 1. And "was this the agent, or the harness?" is NEITHER.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s stop's own instrument

TWO INVOCATIONS, -P codex, -n 2 each, sequential. No acceptance verdict exists and it is
recorded DID NOT RUN, not UNDECIDED and not a pass: the gate is an opencode route and that
provider's weekly limit has stalled §0a row 2 for ten consecutive sessions. Both panels ran
clean (codex ok 62s/63s and 24s/25s), so the findings are results.

  findings/opencode/review-README-20260927T201538Z.md        23136 B, 44 sections
  findings/opencode/review-verify-quotes-20260927T202016Z.md  6555 B,  9 sections

THE INSTRUMENT WAS WRONG, IN FOUR PLACES, ALL THE SAME SHAPE: a failure to fetch reported as
documentation drift — the house failure mode aimed at the tool built to catch it.
  * curl had no --fail: a 404 with a non-empty body exits 0, strips to real text, and every
    quote on that page reports ABSENT at exit 2.
  * an empty CACHED page was accepted where an empty live response or fixture exits 3. Same
    content, two verdicts, decided only by where it came from.
  * an undeclared page key read a missing file with stderr discarded and printed ABSENT, so a
    one-character typo was indistinguishable from real drift.
  * strip keeps page chrome. DISPUTED and registered as a scoped limitation in the header.
PROVED AGAINST THE PRE-FIX SCRIPT rather than asserted: cases J and K each return
`exit 2, found=20 absent=9` at git show HEAD:evidence/p08/verify-quotes.sh. ONE EMPTY CACHED
PAGE AND THE OLD CHECKER REPORTED NINE QUOTATIONS AS STALE.
After the fix: shellcheck exit 0 on both scripts, 14 of 14 fixtures over all four exit codes,
and the live run reproduces found=28 absent=1 exit 2 with the same absent sentence — so the
stop's headline is unchanged by the fix, which is the only reason it may still be stated. The
fail-closed path was OBSERVED, not argued: exit 3 on a page that timed out.

AND THE SAME LAYER MISTAKE AS STOP 22, AGAIN, CAUGHT BY THE SAME REVIEWER AND NOT BY ME:
`staged: true` was L1 and is L2 — the write request is still expressed and something executes
and skips it. So ONE subject row survives step 1, not two. Every count that quoted two is
corrected.

Also fixed from the review: `roles:` does not reach an actorless `schedule:` trigger, which
makes the surface LARGER than the gate's sentence implied; the vocabulary/perimeter claim is
scoped to the documented vocabulary and marked unmeasured for an already-compiled .lock.yml;
"29 network fetches per invocation" was wrong by four-fold (seven pages, 29 sentences);
fail-closed is right for a check whose output GATES A CLAIM, not universally; a redacted
`19:2xZ` identifies no run and now cites an artefact; and the Commit block is annotated as the
deferred labs' deliverables. TWO DISPUTED IN WRITING: the compile-step correspondence check
(the finding restates the section's conclusion as an omission) and the deliberately broad jq
pattern (a superset matching nothing proves the subset matches nothing). FIVE REGISTERED AS
DEBTS of the deferred labs, not fixed — §6 forbids designing a future step's decision rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, and found a fifth defect

THE GATE THAT HAS NOT RUN IN TEN SESSIONS RAN ON THIS ROUTE — lab-acceptance · minimax-m3 —
and BLOCKED. Both line-level runs raised the same thing independently:

  strip() had NO EXIT-STATUS CHECK. `set -uo pipefail` does not catch it (`-e` is absent and
  the status was never read), so a crashing python3, a missing module or ONE NON-UTF-8 BYTE
  left an empty .txt, the matcher read it, and EVERY QUOTE PRINTED ABSENT AT EXIT 2 —
  an internal processing failure reported as documentation drift, in the same format as a
  real finding.

PROVED AGAINST THE PRE-FIX SCRIPT, NOT ASSERTED:
  case L  (a python3 on PATH that exits 1)      OLD exit 2, found=0 absent=29
  case M  (a page with a non-UTF-8 byte)        OLD exit 2, found=0 absent=29
ONE BROKEN INTERPRETER AND THE OLD CHECKER DECLARED THE ENTIRE EXTRACT FABRICATED. The four
round-1 defects each manufactured nine phantom findings; this one manufactures all 29. It is
the sharpest instance of this project's house failure mode found so far — a control whose
failure is indistinguishable from, and louder than, the finding it was built to make.

Fixed: strip failure or empty output -> exit 3. shellcheck exit 0 on both scripts,
16 of 16 fixtures over all four registered exit codes, and the live run still reproduces
found=28 absent=1 exit 2 with the same absent sentence — so the stop's headline is unchanged
by FIVE fixes to the instrument that produced it, which is the only reason it may be stated.

Round 3 not run and the loop closed at two: §4a caps at three, the blocking finding is fixed,
and the gate is a different model from the line-level pass, so a third round would be a third
detection threshold rather than an answer. Two rounds, five defects, all fixed, one finding
(page chrome) disputed in writing and registered as a scoped limitation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous Position block is kept VERBATIM under `## Position — superseded 2026-09-28 at
the stop-23 close`; nothing of it is deleted.

Two headlines, and the second is the sharper one: a verbatim quote in this project's own
August extract went stale in 49 days with nothing executing to catch it — and the instrument
built to catch it had FIVE defects of a single shape, every one reporting a FAILURE TO FETCH
OR PARSE as DOCUMENTATION DRIFT. One empty cached page manufactured nine phantom stale
quotations; one broken interpreter or one non-UTF-8 byte manufactured all twenty-nine.

Also: `staged: true` was L1 and is L2, the same layer mistake as stop 22 and caught by the
same reviewer, so ONE subject row survives step 1 and not two; `roles:` does not reach an
actorless `schedule:` trigger, which makes the surface larger than the exit gate's own
sentence implied; and the vocabulary/perimeter claim is scoped to the documented vocabulary
and marked unmeasured for an already-compiled .lock.yml.

Board check is RED and EXPECTED — the boards are the author's by decision 12 item 4 and are
NOT republished. Required marker digest, re-derived by RUNNING ./tools/check-board-freshness.sh
after the last HANDOFF.md edit rather than by hand: c1e2b847fce9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hermanngeorge15
hermanngeorge15 merged commit c5f461d into main Sep 28, 2026
9 of 10 checks passed
@hermanngeorge15
hermanngeorge15 deleted the stop23/phase-8-agentic-workflows branch September 28, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant