feat(adp): 12-factor-agent Tier U umbrella (W3.1, final Epic #302) - #358
Conversation
…l home (W3.1) Add realizingInClaudeCode Tier U block to 12-factor-agent.ts: 193w openingFraming framing the 12 factors as an umbrella methodology, 24 umbrellaPointers mapping each sibling pattern to its realized factor, and the token-economics closingRule as the canonical home for the meta-rule. Additive edits to context-engineering.ts and memory-management.ts resolve W1.2/W1.4 forward-references by adding 12-factor-agent to seeAlso.siblingPatternSlugs. Updates TOOL-2 script to include 12-factor-agent.ts in the meta-rule consistency check. Final PR of Epic #302. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
julianken-bot
left a comment
julianken-bot
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES — two IMPORTANT findings on the umbrella's authoritative content; one SUGGESTION.
What I verified (this turn)
bash scripts/check_meta_rule_consistency.sh→ exit 0 (OK for context-engineering, memory-management, 12-factor-agent).pnpm test:unit→ 31 files / 420 tests passed; Tier U validators (openingFraming/closingRule/umbrellaPointers≥10/all slugs resolve) all green for 12-factor-agent.pnpm typecheck→ clean.pnpm lint:adp→ typecheck-sketches OK (23 compiled), validate-references OK (45/45), check-affiliate-links OK (265 URLs), lint-changelog OK.- Slug-resolution sweep (Node script over the 24
umbrellaPointers[].patternSlug) → all 24 resolve to files insrc/data/agentic-design-patterns/patterns/. oneLineword-count sweep over all 24 entries → none exceed 20 words.openingFraming= 193w;closingRule= 20w.- Diff scope: 4 files (~170 diff lines). HEAD =
9222a80; base =bedd0e0.
Findings (3 — see inline comments below for detail)
- [IMPORTANT]
12-factor-agent.ts:267—closingRuleparaphrases the user-CLAUDE.md sentence; the schema's type-comment requires "verbatim meta-rule" andcontext-engineering.tsalready quotes the verbatim form. - [IMPORTANT]
12-factor-agent.ts:208—evaluation-llm-as-judgeoneLine cites "Factor 7 adjacent — human-gated merge"; the pattern is LLM-grading-LLM (no humans), and Julian's bot APPROVE flow is LLM-gated, not human-gated. Misrepresents the pattern at catalog level. - [SUGGESTION]
12-factor-agent.ts:236—mcponeLine claims "Factor 4 + Factor 11"; Factor 4 fits cleanly, but Factor 11 ("trigger from anywhere") is about agent dispatch surface, not cross-vendor tool discovery.
Praise
The umbrella's openingFraming reads as a methodology map, not a gap inventory — "each sibling is a production implementation a team assembles, selects among, and owns" is the right framing for a Tier U umbrella and survives R7's anti-example sweep cleanly.
Bottom line: fix needed — both IMPORTANTs concern text that becomes the catalog's authoritative answer the moment Epic #302 closes; better to land them clean than amend later.
Reviewer: @julianken-bot (opus, claude-opus-4-7[1m]) — fresh-context subagent dispatched via the reviewing-as-julianken-bot skill. Verdict above is binding regardless of GitHub's review label. Implementer was sonnet, reviewer is opus → cross-tier OK (R12).
| oneLine: 'Factor 11 — trigger from anywhere: delivers partial output as generated for any consumer transport.', | ||
| }, | ||
| ], | ||
| closingRule: 'If a rule has a trigger (when X, each time X, before/after X), it belongs in a skill, not here.', |
There was a problem hiding this comment.
[IMPORTANT] — closingRule paraphrases the user-CLAUDE.md sentence; schema requires verbatim
Issue: The RealizingInClaudeCode type comment in src/data/agentic-design-patterns/types.ts says: "Tier U only: closing rule sentence (verbatim meta-rule)". The user-CLAUDE.md text reads:
If a rule has a trigger ("when adding screenshots", "before committing", "during PR review"), it belongs in a skill, not here.
This PR writes:
If a rule has a trigger (when X, each time X, before/after X), it belongs in a skill, not here.
The TOOL-2 substring check (it belongs in a skill, not here) passes, but the schema explicitly contracts for verbatim. context-engineering.ts:bodyMarkdown already quotes the original verbatim form including the original three parentheticals — so the catalog now has two competing forms of "the canonical meta-rule," with this PR declaring 12-factor-agent.ts to be the canonical home but storing the paraphrase.
Scenario: A future reader (human or agent) searches the codebase for the canonical meta-rule text, finds two divergent versions in the two pattern files, and has to ask "which is canonical?" The answer should be unambiguous on a Tier U umbrella designed to be the canonical home. As Epic #302 closes, this is the moment to fix it.
Fix: Replace the closingRule value with the verbatim user-CLAUDE.md sentence:
closingRule: `If a rule has a trigger ("when adding screenshots", "before committing", "during PR review"), it belongs in a skill, not here.`,The TOOL-2 substring check still passes (the load-bearing fingerprint is unchanged), and the schema's "verbatim meta-rule" contract is honored.
| }, | ||
| { | ||
| patternSlug: 'evaluation-llm-as-judge', | ||
| oneLine: 'Factor 7 adjacent — human-gated merge: bot APPROVE gate closes the loop with a structured signal.', |
There was a problem hiding this comment.
[IMPORTANT] — evaluation-llm-as-judge mapping misclassifies the pattern as Factor 7 / "human-gated merge"
Issue: The pattern's oneLineSummary (in evaluation-llm-as-judge.ts:8) is "Score model outputs with another LLM applying a written rubric" — fundamentally LLM-grading-LLM, a quality/scoring mechanism. The umbrella's oneLine here cites "Factor 7 adjacent — human-gated merge: bot APPROVE gate closes the loop with a structured signal."
Two problems:
- Factor 7 is "contact humans through tool calls" (per the openingFraming and the 12-factor-agent body). LLM-as-Judge's defining property is that it replaces humans with another LLM —
bodySummary[0]: "LLM-as-Judge replaces human raters with a stronger language model". Citing Factor 7 (even "adjacent") inverts the pattern. - "Human-gated merge: bot APPROVE gate" is factually inverted. The bot APPROVE flow (Identity-Separated Review) is an LLM-gated merge by a separate machine identity; calling it "human-gated" misrepresents both the pattern under discussion (LLM-as-Judge) and the umbrella's adjacent pattern (
identity-separated-review, which on line 252 is correctly labeled "separate machine-user identity runs rubric review before merge").
Scenario: Once this PR merges and Epic #302 closes, the umbrella becomes the catalog's authoritative answer to "which 12-factor does LLM-as-Judge realize?" Future readers — and future agents writing umbrella entries — will treat this mapping as ground truth.
Fix: Either
- (a) re-cite to Factor 9 (compact errors back into the window): "Factor 9 — compact errors back into the window: scores model output against a rubric so regressions surface before they ship." — this matches the rubric/scoring loop;
- (b) drop the pointer entirely if no factor cleanly fits (the 12-factor list is methodology, not a universal taxonomy; eval/quality patterns are slightly orthogonal). The umbrella still hits 23 pointers, well above the ≥22 AC.
| }, | ||
| { | ||
| patternSlug: 'mcp', | ||
| oneLine: 'Factor 4 + Factor 11 — tools as outputs, trigger anywhere: open protocol for cross-vendor tool discovery.', |
There was a problem hiding this comment.
[SUGGESTION] — mcp Factor 11 citation reaches
Issue: "Factor 4 + Factor 11 — tools as outputs, trigger anywhere: open protocol for cross-vendor tool discovery." Factor 4 ("treat tools as structured outputs") is a clean match — MCP defines a JSON-RPC-shaped tool surface. Factor 11 ("trigger from anywhere") is about agent dispatch surface — the agent can be invoked from web request, queue worker, cron, webhook, Slack bot. MCP doesn't speak to triggering; it exposes tools across vendors. Compare with a2a on line 240, where Factor 11 fits cleanly because A2A is literally about cross-runtime agent triggering over HTTP.
Scenario: Same as finding 2 — once merged, this is catalog truth. The misalignment is mild (Factor 4 carries the weight) but visible to a careful reader walking the 12-factor → realization map.
Fix: Drop Factor 11; lead with Factor 4 alone, or pair with the cross-vendor portability lineage:
oneLine: `Factor 4 — tools as structured outputs: open protocol for cross-vendor tool discovery and call.`,Lowest severity because the Factor 4 anchor stands on its own.
F1: replace closingRule with verbatim meta-rule from user-CLAUDE.md.
F2: re-cite evaluation-llm-as-judge to Factor 9 (compact errors back
into the window) — Factor 7 inverted the pattern; Factor 9 maps
honestly to rubric-scoring before ship.
F3: drop Factor 11 from mcp oneLine; Factor 4 alone maps correctly
to cross-vendor tool discovery (Factor 11 fits a2a, not mcp).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Cycle-2 push: 59bcc64 — three fixes from julianken-bot review. F1 [IMPORTANT] — closingRule verbatim F2 [IMPORTANT] — evaluation-llm-as-judge oneLine F3 [SUGGESTION] — mcp Factor 11 drop |
julianken-bot
left a comment
julianken-bot
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — 1 IMPORTANT (cycle-2 review, third pass)
Verification ledger (this turn):
bash scripts/check_meta_rule_consistency.sh→ exit 0; closingRule substringit belongs in a skill, not herepresent (line 267).pnpm test→ 420/420 passed across 31 files.pnpm lint→ 0 errors, 18 pre-existing warnings.pnpm exec tsc --noEmit→ clean.pnpm lint:adp→ typecheck-sketches OK (23 compiled), validate-references OK (45/45), check-affiliate-links OK (265 URLs), lint-changelog OK.git diff 9222a80..59bcc64→ exactly 3 line changes; 24 umbrellaPointers count holds; spot-check of 5 untouched entries (context-engineering,tool-use-react,reflexion,evaluator-optimizer,routing) confirms no regression.closingRuleis now exact-verbatim of/Users/jul/.claude/CLAUDE.mdline 3, including the three parentheticals (F1 cleanly fixed).mcponeLine drops Factor 11; the remaining Factor-4 cite is honest within the umbrella's convention (F3 cleanly fixed).
Findings ToC:
- F2 (line 208, evaluation-llm-as-judge oneLine): cycle-2 fix re-cites Factor 9, but Factor 9 doesn't cover offline CI evaluation. See inline.
What worked well: F1's verbatim restoration (closingRule line 267 now matches the source meta-rule character-for-character, three parentheticals included) is a clean fix — the schema requires verbatim and the fix delivers it. F3's drop of Factor 11 is correct: cross-vendor tool discovery is not what "trigger from anywhere" governs, and the remaining Factor-4 cite is internally consistent with how tool-use-react (line 184) maps to the same factor.
Bottom line: F2 needs one more pass. The cycle-1 issue was a genuinely-misclassified mapping; the cycle-2 fix moves to a different misclassification rather than landing on an honest one. This is the catalog's authoritative factor-to-pattern map and the umbrella exists to make these mappings honest. Empty-findings APPROVE on a pass that ships a known-wrong factor cite would betray the rubric.
Same-tier risk: NO (implementer=sonnet, reviewer=opus).
— @julianken-bot (review cycle 2, R8 third pass)
| }, | ||
| { | ||
| patternSlug: 'evaluation-llm-as-judge', | ||
| oneLine: 'Factor 9 — compact errors back into the window: scores model output against a rubric so regressions surface before they ship.', |
There was a problem hiding this comment.
IMPORTANT — F2 fix re-cites Factor 9, but Factor 9 doesn't cover this mechanic.
Source: HumanLayer Factor 9 (factor-09-compact-errors.md) is in-agent runtime error compaction: catch tool-call exceptions, format them, feed them back to the agent so the agent itself retries with the error visible. The umbrella's own openingFraming (line 168) names it compact errors back into the window paired with keep agents small and focused — operational, in-loop.
LLM-as-judge has no such loop-back to the candidate model. evaluation-llm-as-judge.ts:10 describes it as a CI step ("running thousands of comparisons becomes a CI step rather than a quarterly contract") producing a structured score-and-rationale that humans/CI consume to gate a release. Nothing flows back into a candidate's context window.
Cross-check inside this same file:
- line 204
reflexion: Factor 9 is honest — the agent writes self-critiques into its own memory. - line 212
evaluator-optimizer: Factor 9 is honest — the critic's score feeds the next refinement. - line 208
evaluation-llm-as-judge(this line): borrows evaluator-optimizer's phrasing ("refine until critic stops" → "regressions surface before they ship") without the loop-back mechanic that justified the cite.
Cycle-1 cited Factor 7 adjacent (wrong: inverts LLM-as-judge with human-gated review). Cycle-2 cites Factor 9 (also wrong: no error compaction back into a window). Honest factor candidates:
- Factor 6 (small focused agents) — a stronger judge model is a different specialist than the candidate; the rubric scopes the judge narrowly. The pattern's own
whenNotToUsewarns against same-model self-judging, which is the Factor-6 "different specialist" intuition. - Or keep on Factor 9 with explicit "adjacent" framing that names the asymmetry: judges score outputs but the score is consumed by CI, not fed back to the candidate.
Either is honest; the current oneLine is not.
…ialist, no loop-back) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cycle 3 fix: F2 re-cite evaluation-llm-as-judge to Factor 6Finding: Cycle-2's F2 fix re-cited evaluation-llm-as-judge to Factor 9 ("compact errors back into the window"), but Factor 9 covers in-agent runtime error compaction — catching tool-call exceptions and feeding them back to the agent so it can retry. evaluation-llm-as-judge has no loop-back to the candidate model; the judge's score is consumed by CI/humans to gate a release, not fed back to the pipeline under test. Borrowing evaluator-optimizer's Factor 9 phrasing without that loop-back mechanic was dishonest. Resolution (bot's Option A): Re-cited to Factor 6 — small focused agents. The mapping is honest on two axes:
This also avoids three Factor-9 citations stacking redundantly alongside reflexion (line 204) and evaluator-optimizer (line 212), both of which have genuine loop-back mechanics. New oneLine (line 208): 16 words (≤20 cap). Single-line diff: Gates: Commit: |
julianken-bot
left a comment
julianken-bot
left a comment
There was a problem hiding this comment.
Verdict: APPROVE — empty findings.
Verification ledger
- HEAD:
40e5515— confirmed viagh pr view 358 --json headRefOid. - Diff scope:
git diff 59bcc64..40e5515 --stat→ 1 file / 1 insertion / 1 deletion. The single change is line 208 of12-factor-agent.ts(evaluation-llm-as-judgeumbrellaPointer). F1 (line 267 closingRule), F3 (line 235 mcp), and the other 22 umbrellaPointer entries are byte-identical to cycle 2. - F2 fix: Factor 9 → Factor 6. The new oneLine (17 words, ≤20 cap) maps the pattern to small-focused-agents:
'Factor 6 — small focused agents: stronger judge model, rubric-scoped and distinct from the candidate it grades.' - Cross-check against the cited pattern's own framing:
evaluation-llm-as-judge.tsline 29 (whenNotToUse) says "pick a judge from a different family or a stronger tier", and line 12 calls out "a same-model judge systematically agrees with its own output, so the judge must come from a different family or a stronger tier than what it scores". The new oneLine's "stronger judge model... distinct from the candidate it grades" is a clean compression of that stance. - Internal consistency with the umbrella's own Factor 6 framing: all six Factor 6 entries (lines 192, 196, 208, 220, 244, 256) share the "small focused agents" lexical prefix. Line 208 follows the established pattern.
pnpm test:unit→ 420/420 pass (31 files, 3.69s).bash scripts/check_meta_rule_consistency.sh→ exit 0; all 3 targets (context-engineering, memory-management, 12-factor-agent) carry the canonical fingerprint.- umbrellaPointers count: 24 (
grep -c "patternSlug:" 12-factor-agent.ts= 24, matches AC).
Findings
None. R8 second pass with the explicit prior "find at least one improvement" came up empty. The fix is the minimum surgical change that resolves cycle-2's F2 with semantically correct framing, untouched scope elsewhere, all invariants green.
Specific praise
The cycle-2 verdict offered two honest options: Factor 6 (different specialist) or Factor 9 with explicit "adjacent" framing. Picking Factor 6 — and making the realization concrete ("stronger judge model... distinct from the candidate") rather than abstract — is the choice that maps most cleanly to the cited pattern's whenNotToUse warning about same-model self-judging. The oneLine reads as a mechanism description, not a label.
Bottom line
Final PR of Epic #302 ships clean. Tier U umbrella canonical. Approving.
—
Reviewed by @julianken-bot (model: opus). Implementer ran on sonnet → cross-tier per R12, no same-tier risk.
|
@Mergifyio queue |
Merge Queue Status
This pull request spent 14 minutes 25 seconds in the queue, including 14 minutes 13 seconds running CI. Required conditions to merge
|
2 participants
Diagrams
No visual component changes. This PR adds a data-only
realizingInClaudeCodeTier U block to the 12-factor-agent pattern.Summary
12-factor-agent.ts: AddsrealizingInClaudeCodeTier U block —openingFraming(~193w framing the 12 factors as a methodology umbrella), 24umbrellaPointers(each sibling pattern mapped to its realized factor with a ≤20woneLine), andclosingRule(the token-economics meta-rule verbatim, canonical home for the substringit belongs in a skill, not here).context-engineering.ts: Additive edit — adds12-factor-agenttoseeAlso.siblingPatternSlugs, resolving the W1.2 forward-reference. UpdateslastChangeNote.memory-management.ts: Additive edit — updateslastChangeNoteto confirm W1.4 forward-reference is resolved by the umbrella landing (seeAlso already included12-factor-agent).scripts/check_meta_rule_consistency.sh: Adds12-factor-agent.tsto the TARGETS array per the TODO comment left in W3.1's TOOL-2 prerequisite (infra: add meta-rule consistency check (TOOL-2) + CI workflow #354).Screenshots
N/A — data-only changes.
Test plan
pnpm test:unit— 420 tests pass (31 test files). Tier U invariant tests for12-factor-agent:tier is one of A|B|C|U,seeAlso is present,siblingPatternSlugs all resolve,openingFraming non-empty,closingRule non-empty,umbrellaPointers ≥10,all umbrellaPointers[].patternSlug resolve— all 7 pass.pnpm lint— 0 errors, 18 pre-existing warningspnpm typecheck— cleanpnpm lint:adp— typecheck-sketches OK, validate-references OK, check-affiliate-links OK, lint-changelog OKbash scripts/check_meta_rule_consistency.sh— exit 0 (OK for all 3 targets including new12-factor-agent.ts)git diff | grep -iE "miss\b|gap\b|fail.open|anti.example|caught|the bot found"— 0 hitsposts-listingdate format +theme-aware-herocrossfade) unrelated to data changes; 58 passPlan reference
Action
W3.1indocs/agentic-bridge/reframe/action-plan-v1.json. Final PR of Epic #302 (the ADPrealizingInClaudeCodeepic). TheclosingRulefield in12-factor-agent.tsis now the canonical home for the token-economics meta-rule; all citing patterns reference it viaseeAlso.siblingPatternSlugs.Closes #322