test: extend TestErrorDisplaySync to build_site.py and AGENTS.md mirrors

[chethanuk]

Summary

Extends TestErrorDisplaySync (tests/test_errors.py) to guard the two E001 (missing_contract_block) spec-ref mirrors that were previously unguarded. The canonical E001 diagnostic is duplicated across six places; four were already test-guarded (README.md, docs/index.html, spec/00-introduction.md, plus the canonical vera/errors.py), but scripts/build_site.py and AGENTS.md were not — which is how the §5.4 → §5.2 spec_ref repoint in #826 drifted those two copies before review caught it.

Two extractors and eight assertions close the gap:

• _extract_build_site_error_block — reads the E001 example that scripts/build_site.py renders into docs/index.md and un-doubles the f-string's literal braces, then asserts error_code / description / rationale / fix / spec_ref / header format (6 assertions). The extractor anchors on the closing code fence rather than the spec_ref text, so a future spec_ref drift surfaces the precise field mismatch instead of an opaque "block not found".
• _extract_agents_diagnostic — parses each json fenced block in AGENTS.md, finds the diagnostic whose error_code == "E001", and asserts error_code + spec_ref (the two fields that appear verbatim there; description/rationale are ellipsis-truncated). Assertions use dict.get() so a missing field still shows the helpful message.

Test-only. No drift exists today, so this is a regression guard, not a fix: a future E001 change that misses a mirror now fails a test rather than shipping.

Related Issues

Closes #829

Type of Change

• Tests

How verified

python -m pytest tests/test_errors.py::TestErrorDisplaySync -q → 26 passed (18 → 26; +8 new drift-guard assertions). Touches only tests/, so the CHANGELOG gate does not apply.

Checklist

• I have read CONTRIBUTING.md
• My changes follow the project's coding standards
• I have added/updated tests as appropriate
• I have updated relevant documentation
• All tests pass locally

Summary by CodeRabbit

• Tests
  • Expanded canonical E001 diagnostic validation to cross-check additional rendered and documented outputs, including both formatted text and JSON variants.
  • Added assertions for key fields (error code, description, rationale, fix, and spec reference) and for the expected header formatting.
• Documentation
  • Updated project status and test metrics across README, FAQ, HISTORY, ROADMAP, and TESTING to reflect the latest totals (6,789).
  • Refreshed the TESTING “Test Files” coverage notes for expanded error-display synchronisation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6b754d60-64af-441a-9789-a5a296558796

📥 Commits

Reviewing files that changed from the base of the PR and between 471ff1f and 57a7e06.

⛔ Files ignored due to path filters (1)

• docs/llms-full.txt is excluded by !docs/**

📒 Files selected for processing (6)

• FAQ.md
• HISTORY.md
• README.md
• ROADMAP.md
• TESTING.md
• tests/test_errors.py

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• aallan/vera-bench (manual)

---

📝 Walkthrough

Walkthrough

This PR expands E001 synchronisation tests to cover scripts/build_site.py and AGENTS.md, and updates project status documents to report the newer test totals and revised E001 roadmap note.

Changes

E001 mirror test coverage

Layer / File(s)	Summary
E001 mirror extraction and assertions tests/test_errors.py	Adds helpers for extracting the generated E001 block and the AGENTS.md JSON example, then extends TestErrorDisplaySync to compare the canonical diagnostic fields against both mirrors.

Project status metric updates

Layer / File(s)	Summary
Project status metric updates README.md, FAQ.md, ROADMAP.md, HISTORY.md, TESTING.md	Updates the reported test-count figures and the ROADMAP.md E001 tracking note across the documentation set.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

• Single-source the E001 example so the five doc mirrors cannot drift #954: Matches the new TestErrorDisplaySync coverage for AGENTS.md and scripts/build_site.py mirrors of the E001 diagnostic.

Possibly related PRs

• aallan/vera#327: Introduces scripts/build_site.py, which is one of the E001 mirrors now validated by tests/test_errors.py.
• aallan/vera#526: Affects the generated docs/index.md path that the new build_site.py extraction logic is checking for E001 drift.

Suggested labels: tests, docs

🚥 Pre-merge checks | ✅ 8

✅ Passed checks (8 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly states the main test coverage expansion and names both newly covered mirrors.
Linked Issues check	✅ Passed	The PR adds TestErrorDisplaySync coverage for both AGENTS.md and scripts/build_site.py as requested by #829.
Out of Scope Changes check	✅ Passed	The ancillary documentation metric updates appear consistent with the added test count and do not show unrelated functional changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Changelog Covers Public-Surface Changes	✅ Passed	PASS: the diff only touches docs/tests; no vera/cli.py, vera/errors.py, spec/, vera/lsp/, or vera/codegen/api.py changed, and CHANGELOG.md is untouched.
Spec And Implementation Move Together	✅ Passed	No files under vera/ or spec/ changed; the PR only updates tests and documentation metrics, so there is no implementation/spec drift.
Diagnostics Carry An Error Code	✅ Passed	PR only adds tests/docs; no diagnostic definitions changed. The canonical E001 constructor already carries stable code E001, and W001 is stable too.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.29%. Comparing base (e87a996) to head (57a7e06).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #951   +/-   ##
=======================================
  Coverage   93.29%   93.29%           
=======================================
  Files          96       96           
  Lines       31440    31440           
  Branches      456      456           
=======================================
  Hits        29331    29331           
  Misses       2096     2096           
  Partials       13       13           

Flag	Coverage Δ
javascript	78.41% <ø> (ø)
python	95.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aallan]

Thank you for this — and welcome. It's a genuinely well-made first contribution, and it closes #829 properly.

Because this touches the project's own drift guards, I put it through an adversarial review: several independent passes over the diff, each finding then attacked by separate reviewers instructed to refute it by default, plus a mutation pass in an isolated worktree that actually broke each mirror to check the new tests notice.

The tests are not vacuous — that was verified, not assumed

CLAUDE.md's bar is "a test that is green both before and after the change proves nothing." Your tests clear it. Repointing the canonical spec_ref turns both new mirrors RED alongside the three existing ones, and deleting or corrupting either source fails loudly (assert match / raise AssertionError) rather than silently returning a degenerate value. That last property is the one most sync tests get wrong.

Some things I want to call out specifically, because they're the reasoning this project asks for and rarely gets:

• You anticipated the self-defeating guard. Anchoring _extract_build_site_error_block() on the closing code fence rather than the spec_ref text — with the comment explaining that anchoring on spec_ref would make extraction fail opaquely and hide the very mismatch test_build_site_has_spec_ref exists to surface. I verified this: a spec_ref drift now produces a precise Expected/Got diff, not a "block not found."
• _extract_agents_diagnostic() keys on the parsed payload, not a raw substring — more robust than the three older extractors sitting next to it. For contrast, our own _extract_readme_error_block silently returns '' on a heading drift; both of yours raise with the filename.
• Scope discipline: +96 lines, tests/ only, no compiler surface.

One real gap the mutation pass found

AGENTS.md's fix was unguarded, and the comment explaining why was incorrect.

The comment read "the other fields are JSON-escaped, so only error_code and spec_ref appear verbatim." But _extract_agents_diagnostic() calls json.loads, which resolves the escaping — every field is directly comparable afterwards. (spec_ref is itself escaped in the raw markdown, yet test_agents_has_spec_ref compares it fine, so the reasoning doesn't hold under either reading.) Measured against HEAD, fix deserialises byte-identically to canonical.fix.

Mutation-proven:

Drift AGENTS.md's fix  ('>= 0' -> '>= 999'):   60 passed   <-- GREEN, undetected
Drift the CANONICAL fix ('>= 0' -> '>= 1'):
    FAILED test_readme_has_fix
    FAILED test_html_has_fix
    FAILED test_spec_has_fix
    FAILED test_build_site_has_fix     <-- yours, working
    (no test_agents_has_fix existed)

So AGENTS.md was the one mirror that could go stale while every other was updated in lockstep — exactly the #829 drift class. The wrong comment was the more harmful half: it's the artifact that would have stopped the next person from closing the gap.

Note this is a gap in an area you added coverage to — you guarded fix for build_site.py correctly.

What I pushed to your branch

Maintainer-edit was enabled, so rather than send you round again I pushed two commits:

471ff1f — doc-count lockstep. check_doc_counts.py gates the total test count and the per-file TESTING.md row, so adding tests requires the counts move together. Nothing to do with your code; it's a gate that's easy to miss from outside.

57a7e06 — the review fixes:

• test_agents_has_fix (exact comparison), and the comment replaced with an accurate one.
• test_agents_truncated_fields_are_prefixes — description / rationale are ellipsis-truncated in the example, so they're prefix-compared. This brings AGENTS.md to parity with the other four mirrors.
• Refreshed the TestErrorDisplaySync docstring (it still named 3 of the now-5 mirrors).
• ROADMAP.md: removed Extend TestErrorDisplaySync to cover all E001 spec_ref mirrors (AGENTS.md + the build_site.py generator) #829's row, since this PR closes it.
• TESTING.md: reworded the row (see below).

Both new guards are mutation-validated the same way yours were: drifting AGENTS.md's fix or description turns each RED; reverting returns 62/62 green.

Two notes in fairness

The review caught my mistake, not just yours. In 471ff1f I described the suite as covering "all six E001 spec_ref mirrors" — wrong: #829 says "six places — the diagnostic itself plus five mirrors." The row now enumerates the five instead of asserting a count. I also nearly "corrected" a HISTORY.md snapshot column that turned out to follow a long-standing house convention. Fresh eyes cut both ways.

Not blocking, and not on you: _extract_build_site_error_block() matches the first [E001] in the whole file. There's exactly one today, so it guards the right block and has teeth. A second, earlier example would silently redirect it — but our own pre-existing _extract_html_error_block has the identical shape, so you simply followed house style. Worth hardening both, or neither.

Your ROADMAP row also carried an untracked idea — "single-source the example so nothing can drift" — which I didn't want to delete silently. It's now #954, and the ROADMAP row points there. That's the structural version of what you've built the detection for.

CI is green. Merging. Thanks again — this is exactly the kind of contribution that makes the guardrails worth having, and I hope you'll send more.