Procedure trigger audit: measure which operations procedures fire, then fix the worst

What: Measure how often each operations procedure fires when it should, then fix the worst triggers, starting with wording.
Why: Most procedure triggers are judgments ("substantive prose editing", "material work", "when the action changes") one or two links below the only description the host shows, and nothing tests whether they fire.
So what: An agent builds the trigger table and the baseline from real transcripts now; rewording follows in one PR, and a hook only where rewording measurably fails.

Outcome

A trigger-coverage table for every row of skills/operations/SKILL.md, each with an observable trigger event, the file that must be opened, and its skip rate in real sessions on Claude Code, Codex, Grok and OpenCode. The worst rows get reworded; a re-measurement after a week of real work shows the change.

Evidence (2026-09-29)

Research report on building-agentskills and the AgentsMD writing guides (Opus 5.5, T3 thread "Research: building-agentskills + AgentsMD writing guides"):
- Only Project Direction loading is enforced by mechanism (hook). Every other row depends on the agent noticing a phase change.
- Rated weak: technical-writing -> references/prose.md ("when revising unclear or padded text"), writing-for-agents -> prose.md ("substantive prose editing"), Software design, Artifact placement, Reflection (friction), Finalization, Wayfinder, Repository setup, Reconciliation. Medium: Writing for agents (loses to the Implementation row on an edit), SKILL-MECHANICS.md ("before choosing frontmatter" misses editing an existing description), Verification ("claim readiness" passes unnoticed), Elon method ("material").
- AgentsMD chains references two hops deep (SKILL -> writing-for-agents -> SKILL-MECHANICS or prose); building-agentskills docs/05-authoring/line-budget.md says one level, never nested.
- Existing tests pin wording and links only (tests/test_operations_contract.py); none tests firing.
Research report on obra/superpowers (Opus 5.5, T3 thread "Research: obra/superpowers description guidance"):
- Firing comes from an always-injected bootstrap plus decision-time rationalization tables, not from descriptions. Descriptions are short (92-392 chars), start "Use when", and anchor in time ("before writing implementation code").
- Test method: naive prompts that do not name the skill; deterministic transcript checks (skill called, not called, called before the first Edit/Write); negative over-trigger scenarios; a no-guidance control; five or more runs; the models users actually run (one skill went 5/5 on Opus 4.8 to 0/5 on Opus 5).
This session: the planner skipped SKILL-MECHANICS.md and prose.md while rewriting T3 tool descriptions (Delegation tools are always loaded and name the shell CLIs they replace chromeria#66).
Elon method rarely triggers: gated on Project Direction and tied to artifacts #138 method for the Elon method: 9 of 11 Issue-creating sessions skipped it before rewording, 5 of 13 after.

Elon record

Requirements and who asked: the user (2026-09-29) asked how to verify that every AgentsMD procedure triggers when it should, naming skill mechanics and prose as examples, and agreed to measure first, reword second, and use hooks last.
Deleted: hooks as a first fix; synthetic scenario runs as a release gate (the Objective does not require them); rewriting rows that measure fine.
Bottleneck: no measurement exists, so nobody knows which procedures are dead weight.
Checked myself: read both research reports in full and the routing table rows they cite; reran the Elon method rarely triggers: gated on Project Direction and tied to artifacts #138 count for the Elon method today.

Acceptance criteria

Trigger table: every routing row has an observable trigger event and required file, retained under docs/work/.
Baseline skip rate per row from the last 14 days of real transcripts, per host where transcripts exist, with the counting script retained.
One PR rewords the worst rows: observable "before " triggers, no reference deeper than one hop where a trigger was missed.
Re-measurement one week after release, reported per row.

Non-goals

Hooks for procedures that have not failed after rewording.
Making synthetic model runs a release gate.

Blockers

None.

Required proof

The retained script and both measurements; full suite and versionctl release-check for the rewording PR.

wip: #164 trigger table and baseline skip rates

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

wip: #164 trigger table and baseline skip rates

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

feat: release procedure triggers one hop from routing as 14.2.0 (#164)

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

Baseline, real sessions on the current routing rows (measured 2026-10-01 17:50 UTC). Frozen definition: previous comment. Sessions started 2026-09-30 or later: Claude Code 33, Codex 23, Grok 9, OpenCode 4 (all top-level; 353 Claude and 20 OpenCode sessions under tmp/eval directories dropped). Cells read skipped/trigger sessions.

Row	All hosts	Claude Code	Codex	Grok	OpenCode	2026-09-29 baseline (14 days, fired/trigger)
`entry`	3/20 skip (15%)	2/15 skip (13%)	1/3 skip (33%)	0/1 skip (0%)	0/1 skip (0%)	119/361 (67%)
`elon`	6/7 skip (86%)	3/3 skip (100%)	2/3 skip (67%)	1/1 skip (100%)	-	25/43 (42%)
`elon>algorithm`	6/7 skip (86%)	3/3 skip (100%)	2/3 skip (67%)	1/1 skip (100%)	-	19/43 (56%)
`tickets>decomp`	6/6 skip (100%)	2/2 skip (100%)	3/3 skip (100%)	1/1 skip (100%)	-	5/31 (84%)
`impl`	2/10 skip (20%)	2/9 skip (22%)	-	-	0/1 skip (0%)	87/323 (73%)
`impl>verif`	2/7 skip (29%)	2/7 skip (29%)	-	-	-	56/165 (66%)
`test-design`	1/6 skip (17%)	1/5 skip (20%)	-	-	0/1 skip (0%)	12/227 (95%)
`orch`	8/10 skip (80%)	5/7 skip (71%)	2/2 skip (100%)	1/1 skip (100%)	-	20/115 (83%)
`verif`	3/12 skip (25%)	2/10 skip (20%)	-	1/1 skip (100%)	0/1 skip (0%)	34/105 (68%)
`review`	2/2 skip (100%)	1/1 skip (100%)	1/1 skip (100%)	-	-	1/64 (98%)
`review>method`	2/2 skip (100%)	1/1 skip (100%)	1/1 skip (100%)	-	-	1/64 (98%)
`artifacts`	1/2 skip (50%)	1/2 skip (50%)	-	-	-	3/5 (40%)
`wfa`	2/4 skip (50%)	2/4 skip (50%)	-	-	-	13/32 (59%)
`wfa>mechanics`	1/1 skip (100%)	1/1 skip (100%)	-	-	-	6/15 (60%)
`wfa>prose`	1/4 skip (25%)	1/4 skip (25%)	-	-	-	3/32 (91%)
`tw`	1/1 skip (100%)	-	-	-	1/1 skip (100%)	4/76 (95%)
`tw>prose`	1/1 skip (100%)	-	-	-	1/1 skip (100%)	2/76 (97%)
`vc`	9/15 skip (60%)	9/14 skip (64%)	-	-	0/1 skip (0%)	95/186 (49%)
`delivery-profile`	1/1 skip (100%)	1/1 skip (100%)	-	-	-	14/77 (82%)
`delivery`	11/12 skip (92%)	9/10 skip (90%)	-	1/1 skip (100%)	1/1 skip (100%)	31/112 (72%)
`final`	4/4 skip (100%)	2/2 skip (100%)	1/1 skip (100%)	-	1/1 skip (100%)	15/72 (79%)

Rows with no trigger in the window are left out (pd, grilling, domain, wayfinder, spec, orch>bounded, proj-verif, reflection, vc>bump, delivery>final, repo-setup, grok).

Worst rows, by the frozen rule (8 or more triggers):

delivery: 11/12 skipped. At gh pr create agents read verification.md (9/12 fired) but not delivery.md, which holds the PR-body rules (delivery.md line 34: "Before opening or updating a PR, read CONTRIBUTING.md ... Open the PR body with the core's Human summary"). The row says "Deliver a PR, merge, release"; it never names opening one.
orch: 8/10 skipped. The row "Delegate or coordinate dependencies, recover a handoff" never names spawning an agent or thread.
vc: 9/15 skipped, mostly Claude Code (9/14). The row "Change tracked files or make an explicit WIP checkpoint" never names committing.

The verif (25%), impl (20%) and entry (15%) rows improved sharply since #174 and #182 (they were at 68%, 73% and 67%).

Case study (correcting my previous comment): session 86d43ea9 made 18 file edits, about 29 git commit calls, 9 gh pr create and 4 gh issue create, and researched and recommended a monitoring design. Its only Skill call was update-config; it never opened operations. The session-start pointer reached it on 3 of its roughly 30 hook deliveries (the SessionStart:resume events). Every later user turn started a new kind of action with no reminder: new tasks arrive as user turns, not as session starts. The single-prompt harnesses (#174, #182) cannot see this, because their pointer is always fresh.

Limits. The window is about 1.5 days, so samples are small; Codex, Grok and OpenCode have 1 to 3 triggers per row. Most Claude triggers come from a few long T3 sessions. elon triggers include false positives: gh issue create matched inside a Python transcript reader and inside a gh pr comment body.

Next: reword the delivery, orch and vc triggers as "before " clauses, the shape #182 proved. Measure before and after with a multi-turn run on all four harnesses, with the pass condition frozen here before it runs.

Before/after measurement, frozen before running (2026-10-01). Pass condition and scenario are fixed here; nothing below changes after results come in.

Instrument: docs/work/164-trigger-audit/multiturn.py at ee9c93d (sha256 f8266d5f…69b21). It reuses trigger_test.py's confinement, real plugin loading per host, fixture repository, gh stub and read scoring. New: one session, five user turns, continued the way each host continues a session (Claude Code: one claude -p --input-format stream-json process, a new user message per turn, so SessionStart fires once, as in T3; Codex exec resume --last; Grok -p --continue; OpenCode run --continue). Each routed action is scored against reads made before it anywhere in the session.

Turns (prompts never name a procedure):

question: what divide() does when b is 0, from the code only
research: research error-reporting options for a small Python library and recommend one, change nothing; scored research/index.md before the turn ends
commit: make divide() raise ValueError, then commit; scored implementation.md before the first edit and version-control/index.md before git commit
pr: open a PR with gh pr create; scored verification.md and delivery.md before gh pr create
delegate: have a separate agent review the change; scored orchestration.md before the first spawn (Agent/Task/spawn_agent/spawn_subagent/task tool, or a shell codex exec/claude -p/opencode run/grok -p)

Arms: main = 8cad98f (origin/main), branch = 2b04650 (the three reworded rows only), none = no-guidance control (login only: no rules file, no plugin). Arms interleaved.
Harnesses: Claude Code claude-opus-5-5 effort medium; Codex gpt-6.1-sol medium; Grok grok-4.7 medium; OpenCode opencode/muse-spark-1.3-contributor-free (default variant). 5 valid runs per arm and host. A run is invalid when fewer than 5 turns ran or it made no tool call; invalid runs are kept in the records and replaced, at most 3 replacements per cell.
Canaries (passed, before this freeze): every host, every arm: main and branch quote # Global Agent Rules and see the operations Skill; Claude Code and OpenCode also see the session pointer; none sees neither.

Pass condition:

P1, per reworded row and host (version-control/index.md at commit, delivery.md at gh pr create, orchestration.md at spawn): branch reads the file before the action in at least 4 of the runs where the action happened, with at least 3 such runs; otherwise the cell is "not measurable". A row ships when it passes on every measurable host.
P2, no over-trigger: on branch, a reworded file opened during turns 1-2 in no more runs than on main, per host.
P3, no regression: research/index.md, implementation.md and verification.md on branch at most 1 run below main, per host.
P4, discrimination: the none control fires 0 everywhere.

Decision rule: a row that fails P1 on a host where operations was never loaded in the session is a candidate for a hook. Any hook gets its own frozen measurement before it ships. Records (per run: call list up to the end, anonymized paths, verdicts, usage) go to docs/work/164-trigger-audit/multiturn.jsonl.

wip: pin the delivery, orchestration and version-control triggers (#164)

feat: name the action in the delivery, orchestration and version-cont…

Before/after result, PR #200 open (2026-10-01). The three worst rows are reworded as "before " in #200 (v14.6.1). Results: results-2026-10-01.md, records multiturn.jsonl on the branch.

Frozen pass condition applied as frozen:

P1: passes for Delivery, Orchestration and Version control on Claude Code (delivery 2/5 to 5/5) and Grok (orchestration 0/5 to 5/5, delivery 4/5 to 5/5), and for Delivery and Version control on Codex (5/5 on both arms). Not measurable: Codex orchestration, because codex exec --json does not emit the spawn (post hoc, orchestration.md was read before the first wait in 10/10 runs), and OpenCode.
P2: 0 over-trigger runs on every host and arm.
P3: no regression.
P4: the control read nothing.

Not shown: the five-turn session does not reproduce the real Claude Code skips of version control and orchestration (main already 5/5). Those rows need the real-session re-measurement one week after release (acceptance criterion 4), with the same measure.py --since <release date>.

Open:

OpenCode free route returns Rate limit exceeded. A retry loop launches the replacement runs (main 2, branch 3, control 3) when it clears; their result goes here.
Independent review of 828e89a is running (review/independent pending).
No hook was built: rewording passed wherever it was measurable.

Handoff: branch feat/164-routing-firing, worktree agentsmd-worktrees/164-routing-firing, head 828e89a, base 8cad98f; python3 -m unittest discover -s tests 396 OK; bin/versionctl release-check OK for v14.6.1. Merge is the user's.

wip: count only reads that returned the file, only completed turns; k…

Re-measurement after independent review, frozen before running (2026-10-01). The review of #200 found three blocking defects, all confirmed. The first run's results are withdrawn as evidence; it stays in the history as run 1.

Fixes, at c0e91ab (multiturn.py sha256 793672072a9a2579fd561463ab84e0eeb4cbf335a98d76b89790075ea0c7e306):

A read counts only when the tool returned the file: the call names the file and its own result text contains the file's first heading. This rejects failed Codex cats, echo cat <path> and refused reads. Checked on a direct read on Claude Code, Codex and Grok; OpenCode could not be checked (rate limit). Pinned in tests/test_trigger_harness.py.
A run is valid only when all five turns completed: Claude Code's per-turn result without an error; elsewhere exit 0 before the watchdog plus the host's end-of-turn event. Stalled turns no longer count.
The version-control row keeps its pre-edit trigger: "Before editing tracked files, committing, bumping a version, or making an explicit WIP checkpoint". The procedure requires versionctl doctor before edits.

Arms: main 8cad98f, branch c0e91ab, none. Scenario, models, 5 valid runs per cell, at most 3 replacements, and pass conditions P1 to P4 are unchanged from the earlier freeze. Codex orchestration stays "not measurable" (no spawn event). OpenCode runs when its free route stops returning Rate limit exceeded. Records: multiturn-r2.jsonl.

wip: multiturn parser skips events whose message is a string (#164)

docs: re-measured before/after under the reviewed scorer (#164)

wip: a read counts only when its output arrived before the action; Gr…

Round 3, frozen before running (2026-10-01). The re-review of #200 found two more scorer defects and one control limitation, all confirmed. Round 2 (multiturn-r2.jsonl) is withdrawn as evidence: its records keep call order but not when outputs arrived, so they cannot be rescored.

Fixes, at afac243 (multiturn.py sha256 6baeef7c…46836683), pinned in tests/test_trigger_harness.py:

A read counts before an action only if its output arrived before the action was invoked, using stream event positions: Claude and Grok tool_use against tool_result, Codex item.started against item.completed. OpenCode reports a tool part once, when it finishes, so for OpenCode both positions are that event.
Turn completion is parsed from JSON: Grok needs a final result event with is_error: false, Codex needs turn.completed, OpenCode needs step_finish. Whitespace no longer matters.
The control arm is scored against main's real procedure headings, so an unexpected successful read would count. The control host still loads nothing.

Arms: main 8cad98f, branch c0e91ab (row text unchanged since round 2), none. Scenario, models, 5 valid runs per cell, at most 3 replacements, and P1 to P4 are unchanged from the first freeze. Records: multiturn-r3.jsonl. OpenCode runs when its free route answers.

wip: a help lookup (gh pr create --help) is not the action (#164)

Round 3 result under its frozen rule, and a Grok-only rerun frozen before running (2026-10-01).

Round 3 (multiturn-r3.jsonl, instrument afac243), main / branch, file read before the action:

Claude Code: delivery 2/5 / 5/5, orchestration 4/5 / 5/5, version control 5/5 / 5/5. P1 passes.
Codex: delivery 5/5 / 5/5, version control 5/5 / 4/5. P1 passes; orchestration not measurable.
Grok: orchestration 1/5 / 5/5 and version control 5/5 / 5/5 pass. Delivery 4/5 / 3/5 fails P1 (needs 4).
P2: 0 over-triggers. P3: within 1 run everywhere (Codex implementation 5/5 / 4/5, verification 5/5 / 4/5). P4: the control read nothing.

Cause of the Grok delivery result: in 12 of Grok's 15 runs (every arm), the first match for gh pr create was gh pr create --help, a help lookup issued alongside the read. The real gh pr create came later. Both branch "misses" and main's one miss are such runs; in each, delivery.md had been read before the real call. No Claude Code or Codex record has a --help match, so their cells cannot change.

Fix at e66bd84 (multiturn.py sha256 4fe0dd26…9aa7dfa7): a shell command performs the action only if the matched command segment does not ask for --help/-h (quoted words ignored). Pinned in tests/test_trigger_harness.py. Rerun: Grok only, arms main 8cad98f / branch c0e91ab / none, 5 valid runs each, same scenario, model and P1 to P4. Its result replaces Grok's round-3 cells; records go to multiturn-r3-grok-rerun.jsonl. Round 3 stays the record for Claude Code and Codex.

docs: round-3 before/after, Grok rerun blocked by its usage balance (#…

State (2026-10-01 23:30 CEST). PR #200 head 39b71ec, v14.6.1. 403 tests OK, release-check OK. Third independent review running.

Round 3 under its frozen rule:

Claude Code: passes all three rows (delivery 2/5 to 5/5, orchestration 4/5 to 5/5).
Codex: passes delivery and version control.
Grok: passes orchestration (1/5 to 5/5) and version control. Fails delivery (3/5), because the instrument took gh pr create --help as the PR action; fixed in e66bd84.
Everywhere: P2 to P4 pass.

Details: results.

Blockers (external):

Grok: 402 Payment Required: Grok Build usage balance exhausted. The fixed Grok rerun stopped at 2 valid runs per arm, below the 3 P1 needs. Topping up the balance is the user's call; after that, rerun with go.sh-equivalent python3 multiturn.py run --host grok --model grok-4.7 --effort medium --arms main=8cad98f,branch=c0e91ab,none --runs 5.
OpenCode: the free route muse-spark-1.3-contributor-free has returned Rate limit exceeded since about 19:00 UTC. Same command with --host opencode --model opencode/muse-spark-1.3-contributor-free --jobs 3.

Still open in #164: re-measure on real sessions one week after release (measure.py --since <release date>), especially version control, which the five-turn session cannot reproduce.

Navigation Menu

Repository navigation

Procedure trigger audit: measure which operations procedures fire, then fix the worst #164

Outcome

Evidence (2026-09-29)

Elon record

Acceptance criteria

Non-goals

Blockers

Required proof

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Procedure trigger audit: measure which operations procedures fire, then fix the worst #164

Description

Outcome

Evidence (2026-09-29)

Elon record

Acceptance criteria

Non-goals

Blockers

Required proof

Activity

6 remaining items

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions