Skip to content

Commit 1d5cd26

Browse files
committed
Fix stale verification claims and drop phrase-order doc tests
docs/lessons-learned.md said "54 tests today cover 9 contract families"; tests/test_marginalia_geometry.py has 31 tests in 13 contract classes. The sentence now points at the classes instead of quoting a number that drifts. It also said the golden fixture is "refreshed explicitly by scripts/refresh_golden_fixture.py"; the script and fixture were deleted in 0929c25, so it now names what catches loader and parser regressions today. tests/test_markdown_migration_prereqs.py checked that the finished migration's spec contained particular phrases, including "Red", "Green" and "Refactor" in that order, which proves nothing about the process. Those tests are dropped; the README and CI command contract stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012eisRQmQvg4TphwztxhQGJ
1 parent 49cf347 commit 1d5cd26

2 files changed

Lines changed: 7 additions & 73 deletions

File tree

‎docs/lessons-learned.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ This document records project lessons that should guide future changes to Python
4040
- Keep the complete editable program visible because it is the thing that actually runs.
4141
- When storing examples as Markdown, keep the full editable program in `:::program` and teaching fragments in separate `:::cell` blocks. Do not concatenate cells to recreate the editor source.
4242
- Fine-grained cells can restate definitions to stay executable. This is better than collapsing class, property, recursion, match, or type-hint examples into one large cell.
43-
- Preserve a frozen golden catalog while migrating source formats. Full stdout parity is not enough; rendered teaching-cell structure must also match. (After the migration, the fixture's job changed: it is now a reviewed structural snapshot, refreshed explicitly by `scripts/refresh_golden_fixture.py`, that catches loader/parser regressions rather than guarding a migration.)
43+
- Preserve a frozen golden catalog while migrating source formats. Full stdout parity is not enough; rendered teaching-cell structure must also match. (After the migration the frozen fixture and its refresh script were retired in 0929c25, 2026-06-13; loader and parser regressions are now caught by the frontmatter property tests in `tests/test_parser_properties.py` and the byte-exact output check in `scripts/verify_examples.py`.)
4444
- Beware broad-surface titles that quietly teach only one narrow slice. Pages named `Testing`, `Packages`, `Regular Expressions`, `Type Hints`, `Async Await`, or `Special Methods` must either cover the forms a reader reasonably expects or explicitly frame themselves as a first pass and link to focused neighbors.
4545
- Do not let an important syntax form live only in a separate page if another page title strongly implies it. An umbrella `Operators` page should at least point to and lightly show `:=`, even though `Assignment Expressions` remains the focused lesson.
4646
- Journey order should follow prerequisite thinking, not catalog order. Put booleans before truthiness and conditionals, scope before closures, bytes before networking, and environment boundaries before subprocess/thread boundaries.
@@ -109,7 +109,7 @@ git diff --check
109109
- **Score what's shipping, not what was designed.** A scoring dict on the gestalt is design-time review. Production figures live in `src/marginalia.py` `FIGURES` and may have been redesigned during promotion. Scoring should track the production version with the gestalt as separate history.
110110
- **Semantic diagram review has four objects: page, cell, paint function, caption.** Geometry contracts can prove that a figure renders, but not that it still teaches the adjacent cell. On every diagram pass, read the current example cell, the attached paint function, the caption, and the score rationale together. The same audit caught `container-protocols` still showing `iter()/next()` after the page had shifted to `__setitem__`/`__contains__`/`__getitem__`, `structured-data-shapes` over-focusing on `TypedDict` while anchored to a dataclass cell, and `object-lifecycle` mentioning `__del__` after the lesson had been reframed around references.
111111
- **Some examples should never have figures.** Constraint-shaped, infrastructure-shaped, and aggregator-shaped slugs lack a single mechanism to depict. Force-fitting figures on them scores below the gate. Leave them figure-less and document why rather than ship weak figures.
112-
- **Audits without contracts rot; bug classes need automated gates.** When you find a bug class — clipping, collision, off-palette colour, drifting twin coordinates, duplicate caption — write a unit test that asserts the invariant across every figure. The geometry contracts in `tests/test_marginalia_geometry.py` started as ad-hoc scripts and were promoted to CI gates after the same bug class recurred. 54 tests today cover 9 contract families; new figures pass them automatically because each test iterates `FIGURES`.
112+
- **Audits without contracts rot; bug classes need automated gates.** When you find a bug class — clipping, collision, off-palette colour, drifting twin coordinates, duplicate caption — write a unit test that asserts the invariant across every figure. The geometry contracts in `tests/test_marginalia_geometry.py` started as ad-hoc scripts and were promoted to CI gates after the same bug class recurred. Each contract family is a test class there; new figures pass them automatically because each test iterates `FIGURES`. (Count the classes rather than quoting a number here: an earlier "54 tests, 9 families" claim drifted from the file.)
113113
- **Clipping ≠ collision; padding the viewBox fixes one but not the other.** The `value-types` bug had two components: the first `INT` tag was clipped above the viewBox (geometry escapes its frame), and the `STR`/`LIST`/`DICT` tags overlapped the boxes above them (geometry collides internally). Padding the viewBox solved (1) and disguised (2). Element-element collision needs its own audit that walks every text-rect, text-text, and (for label-on-edge cases) text-line bounding-box pair.
114114
- **Heuristic audits over-flag; trust the design, not the regex.** Probes for "prose duplication" (SVG text matching caption substring), "text crossing a line" (label bbox bisected by a hairline), and "text overlapping a circle" each surfaced ~3-10 hits — all false positives. Diagrammatic labels naturally appear in captions (`__getattr__`, `yield from inner`); dashed strikes through `.append` are deliberate; text inside node circles is the design. Don't promote heuristic findings to contracts without confirming each is a real bug.
115115
- **Structural twins must share coordinates exactly.** When two figures depict parallel concepts — `kw-only-separator` and `positional-only-separator`, `class-triangle` and `metaclass-triangle` — they read as a pair. A single-pixel drift in one breaks the visual rhyme. Treat a coordinate change in one as a forced change in both, in the same commit. The audit caught `kw-only-separator` at the old `x=82` after `positional-only-separator` had moved to the corrected `x=75`.

‎tests/test_markdown_migration_prereqs.py‎

Lines changed: 5 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -2,85 +2,19 @@
22
from pathlib import Path
33

44
ROOT = Path(__file__).resolve().parents[1]
5-
SPEC = ROOT / "docs" / "example-source-format-spec.md"
65
README = ROOT / "README.md"
76
CI = ROOT / ".github" / "workflows" / "verify.yml"
87

98

109
class MarkdownMigrationPrereqTests(unittest.TestCase):
11-
def test_spec_covers_all_fourteen_migration_prerequisites(self):
12-
spec = SPEC.read_text()
13-
required_phrases = [
14-
"Cloudflare bundling behavior",
15-
"Historical golden parity script",
16-
"Cell model",
17-
"Generated full code",
18-
"Verifier correctness",
19-
"Line-number reporting",
20-
"Formatter behavior",
21-
"Build step",
22-
"Cache busting",
23-
"Worker runtime startup",
24-
"POST execution",
25-
"Browser layout",
26-
"Production smoke test",
27-
"Python version migration command",
28-
]
29-
for phrase in required_phrases:
30-
with self.subTest(phrase=phrase):
31-
self.assertIn(phrase, spec)
10+
# The Markdown migration shipped (see docs/example-source-format-spec.md).
11+
# Tests that only checked the spec's wording, including the order of the
12+
# words "Red", "Green" and "Refactor", were removed: they proved nothing
13+
# about the process. What remains is the contributor-facing contract.
3214

33-
def test_spec_defines_red_green_refactor_order(self):
34-
spec = SPEC.read_text()
35-
self.assertIn("Red", spec)
36-
self.assertIn("Green", spec)
37-
self.assertIn("Refactor", spec)
38-
self.assertLess(spec.index("Red"), spec.index("Green"))
39-
self.assertLess(spec.index("Green"), spec.index("Refactor"))
40-
41-
def test_spec_blocks_deploy_without_full_teaching_parity(self):
42-
spec = SPEC.read_text()
43-
self.assertIn("100% golden parity", spec)
44-
self.assertIn("100% parity", spec)
45-
self.assertIn("Do not deploy Markdown-backed examples unless Phase 3 has 100% parity", spec)
46-
self.assertIn("No allowlist is permitted for the app switch", spec)
47-
self.assertIn("teaching-structure difference", spec)
48-
49-
def test_spec_names_examples_that_lost_fine_grained_cells(self):
50-
spec = SPEC.read_text()
51-
for slug in [
52-
"match-statements",
53-
"recursion",
54-
"classes",
55-
"properties",
56-
"special-methods",
57-
"type-hints",
58-
]:
59-
with self.subTest(slug=slug):
60-
self.assertIn(slug, spec)
61-
62-
def test_spec_captures_successful_migration_and_rollback_rehearsal(self):
63-
spec = SPEC.read_text()
64-
self.assertIn("Lessons from the successful parity migration", spec)
65-
self.assertIn("100% golden parity", spec)
66-
self.assertIn("rollback-rehearsed", spec)
67-
self.assertIn("revert the migration, deploy the rollback", spec)
68-
self.assertIn("re-apply the migration and repeat verification", spec)
69-
70-
def test_remaining_operational_lessons_are_documented_and_verified(self):
71-
spec = SPEC.read_text()
15+
def test_readme_and_ci_keep_the_markdown_workflow_commands(self):
7216
readme = README.read_text()
7317
workflow = CI.read_text()
74-
for phrase in [
75-
"Golden fixture cleanup policy",
76-
"CI policy",
77-
"Contributor documentation policy",
78-
"Native Markdown bundling remains unproven",
79-
"reserialize it deterministically",
80-
"TOML editorial registries",
81-
]:
82-
with self.subTest(phrase=phrase):
83-
self.assertIn(phrase, spec)
8418
for phrase in ["src/example_sources/", ":::program", ":::cell", "make build", "make verify-examples"]:
8519
with self.subTest(readme=phrase):
8620
self.assertIn(phrase, readme)

0 commit comments

Comments
 (0)