ai-forge maintenance: evidence and test hygiene

[dsmcewan]

Scope

Canonical bounded maintenance sweep for the deferred ai-forge product. This consolidates issues #37 and #46 plus the mechanical evidence-hygiene items formerly tracked in #30. These are non-blocking defects and diagnostics improvements; the current tests and gate remain green.

ai-forge remains AM-40-deferred. Implementation must use a newly authorized bounded maintenance plan rather than borrowing the completed Clotho authz-008 scope.

Checklist

• Bind the stored RAG scorecard to the executable evaluation. In ai-forge/patterns/rag.mjs, generated rag/evals/run.mjs must read scorecard.json and fail if stored precision, faithfulness, dataset size, topK, or thresholds differ from recomputed/configured values beyond a documented epsilon.
• Guard all four FC2 DESIGN parsers. Replace direct md.match(...)[1] access in test-{telos,multiagent,eval,serving}-forge.mjs with an explicit missing-block failure carrying a stable, legible message.
• Replace the tautological eval assertion. In ai-forge/patterns/eval.mjs, replace a.length === DATASET.length with a discriminator that can fail, such as proving every prediction belongs to the closed output set.
• Clean generated TELOS self-test temporary directories. Wrap both mkdtempSync uses in ai-forge/patterns/telos.mjs with deterministic finally cleanup using rmSync(..., { recursive: true, force: true }).
• Pin the design heading contract locally. In ai-forge/scripts/test-design.mjs, require the exact ## heading level enforced by design-verify.mjs.
• Make design workstream signatures consistent. Declare render(ctx) and checks(ctx) even though the closed-over implementation does not consume ctx yet.
• Make generated index bytes deterministic. Align rag/index.build.mjs with the renderer by writing the same trailing newline to index.json.

Acceptance

• Each behavioral fix is introduced test-first and observed failing for the intended reason.
• npm --prefix ai-forge test exits 0.
• node docs/institutional-memory/verify-contracts.mjs exits 0.
• git diff --check is clean.
• No dependency, shared-package, spine, enrollment, or mythology change.

Out of scope

The shared forge-kit/ extraction remains separately tracked in #30 because it is an architectural/package-boundary decision, not a mechanical maintenance fix.

[dsmcewan]

@claude take care of th9s please

[claude]

Claude finished @dsmcewan's task in 6m 31s —— View job • claude/issue-63-20260726-1725 • Create PR ➔

---

Done

Todo list

• Bind the stored RAG scorecard to the executable evaluation (ai-forge/patterns/rag.mjs)
• Guard all four FC2 DESIGN parsers (test-{telos,multiagent,eval,serving}-forge.mjs)
• Replace the tautological eval assertion in ai-forge/patterns/eval.mjs
• Clean generated TELOS self-test temp dirs in ai-forge/patterns/telos.mjs
• Pin the design heading contract locally in ai-forge/scripts/test-design.mjs
• Make design workstream signatures consistent (render(ctx) / checks(ctx))
• Make generated index bytes deterministic in rag/index.build.mjs
• Commit and push

Summary

Note: the two checklist boxes already checked in the issue body ("Bind the stored RAG scorecard…" and "Guard all four FC2 DESIGN parsers…") were not actually reflected in the code — grep confirmed rag/evals/run.mjs never read scorecard.json, and all four test-*-forge.mjs files still did raw md.match(...)[1]. I implemented the full checklist (all 7 items) rather than trusting the checkbox state.

• ai-forge/patterns/rag.mjs: rag/evals/run.mjs now reads rag/evals/scorecard.json and fails closed if stored precision, faithfulness, n (dataset size), topK, or thresholds drift from the recomputed/configured values beyond EPS = 1e-9. Also aligned rag/index.build.mjs's index.json write with the renderer's trailing \n so re-running the build script reproduces byte-identical output.
• test-{telos,multiagent,eval,serving}-forge.mjs: the FC2 "drift the design" cases now check the fenced ```json match explicitly and throw a stable, legible error (e.g. test-telos-forge.mjs: no fenced \``json component block in DESIGN.md) instead of crashing on null[1]`.
• ai-forge/patterns/eval.mjs: the target self-test no longer asserts a.length === DATASET.length (always true by construction of .map); it now proves every prediction belongs to the closed {positive, negative} output set.
• ai-forge/patterns/telos.mjs: both generated self-test scripts (ledger, breakout) now wrap their mkdtempSync temp dirs in try { ... } finally { rmSync(dir, { recursive: true, force: true }) }.
• ai-forge/scripts/test-design.mjs: section-heading assertions now require the exact ## level (matching design-verify.mjs's own regex), not any #+ level.
• ai-forge/workstreams/design.mjs: render/checks now declare (ctx) for signature consistency with every other workstream (which generators.mjs always calls with a real ctx), even though the closed-over implementation doesn't consume it yet.

git diff --check is clean.

⚠️ Could not run the test suite or verify-contracts.mjs in this session — every node/npm invocation (including node --check, npm test, npm test --prefix ai-forge) was rejected with "This command requires approval" regardless of form, while git commands executed normally. There's no interactive approver in this automated run, so I couldn't get npm --prefix ai-forge test or node docs/institutional-memory/verify-contracts.mjs executed to confirm exit 0 as the acceptance criteria require. I verified each change by careful manual reading of the surrounding code/tests (confirmed no test hardcodes the old tautological/loose assertions I replaced), but please re-run both commands locally or in CI before merging, or update --allowedTools to permit node/npm so I can verify directly.