Skip to content

Commit 1c01bbb

Browse files
authored
Merge pull request #20 from adewale/claude/verification-audit-fixes
Measure rendered styles in the browser, smoke-test deploys, warn before waivers expire
2 parents 2ab5b23 + 8eb6500 commit 1c01bbb

9 files changed

Lines changed: 506 additions & 290 deletions

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ make check-generated
6666

6767
## Quality checks
6868

69-
These scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`.
69+
These scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`. CI does not schedule a waiver-expiry run; before a release or once a month, run `make check-waiver-expiry`, which fails while any waiver has 30 days or fewer left.
7070

7171
| Script | What it gates |
7272
| --- | --- |
@@ -79,7 +79,7 @@ These scripts enforce the catalog-level rules from `docs/example-quality-rubric.
7979
| `scripts/check_prose_duplication.py` | No verbatim repeated paragraphs, no cell prose copied from the intro, no duplicate note bullets. |
8080
| `scripts/check_inline_links.py` | Inline `[text](target)` links in prose resolve to real `/examples` or `/journeys` pages. |
8181
| `scripts/score_example_criteria.py` | Heuristic criterion scores per example; fails when a curated score exceeds the heuristic by more than the delta bound, so the score registry cannot inflate without the page changing. |
82-
| `scripts/check_quality_scores.py` | Curated scores meet the 9.0 target / 8.5 hard minimum, waivers carry future expiry dates and are dropped when stale, and the journey-section average stays above its floor. |
82+
| `scripts/check_quality_scores.py` | Curated scores meet the 9.0 target / 8.5 hard minimum, waivers carry future expiry dates (with a warning in their last 30 days) and are dropped when stale, and the journey-section average stays above its floor. |
8383
| `scripts/check_no_figure_rationales.py` | No-figure opt-outs carry a reason and an unexpired `review_after` date. |
8484
| `scripts/check_journey_outcomes.py` | Every journey section declares learner outcomes. |
8585
| `scripts/audit_example_graph.py --check` | The `see_also` graph has no broken targets, self-links, over-linked pages, or orphaned examples. |
@@ -97,7 +97,7 @@ The single source of truth for the registries is `docs/quality-registries.toml`.
9797

9898
## Secrets and deploy configuration
9999

100-
Production deployment is manual through an authenticated Wrangler session with `make deploy`; public `workers.dev` and version preview URLs are disabled. Runtime Worker secrets (`TURNSTILE_SECRET_KEY`, `TURNSTILE_CLEARANCE_SECRET`, `PBE_SMOKE_BYPASS_SECRET`) are managed with `wrangler secret put`; see `docs/turnstile-runner-protection-spec.md`.
100+
Production deployment is manual through an authenticated Wrangler session with `make deploy`, which smoke-tests the deployed origin afterwards and fails if the smoke fails; public `workers.dev` and version preview URLs are disabled. Runtime Worker secrets (`TURNSTILE_SECRET_KEY`, `TURNSTILE_CLEARANCE_SECRET`, `PBE_SMOKE_BYPASS_SECRET`) are managed with `wrangler secret put`; see `docs/turnstile-runner-protection-spec.md`.
101101

102102
Generated output is prevented from drifting before merge: install the local hooks with `scripts/install-git-hooks.sh`, and keep `main` protected so pull requests require the `verify` status check to pass against the current base. CI enforces the same `make check-generated` contract for contributors without local hooks.
103103

‎Makefile‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
PY := uv run --python 3.13
44
NODE_DEPS_STAMP := node_modules/.package-lock.json
55

6-
.PHONY: check-node-version node-deps test embed-examples embed-editorial-registry build-search-index build check-generated fingerprint prototypes browser-layout-test search-ranking-test social-cards check-social-cards seo-cache-lint verify-examples check-registry-integrity check-confusable-pairs check-broad-surface-tours check-footgun-coverage check-notes-supported check-program-covers-cells check-prose-duplication check-inline-links score-example-criteria check-quality-scores check-no-figure-rationales check-journey-outcomes audit-example-graph quality-checks rubric-audit format-examples verify-python-version verify smoke-deployment dev deploy upgrade-runtime-deps lint
6+
.PHONY: check-node-version node-deps test embed-examples embed-editorial-registry build-search-index build check-generated fingerprint prototypes browser-layout-test search-ranking-test social-cards check-social-cards seo-cache-lint verify-examples check-registry-integrity check-confusable-pairs check-broad-surface-tours check-footgun-coverage check-notes-supported check-program-covers-cells check-prose-duplication check-inline-links score-example-criteria check-quality-scores check-waiver-expiry check-no-figure-rationales check-journey-outcomes audit-example-graph quality-checks rubric-audit format-examples verify-python-version verify smoke-deployment post-deploy-smoke dev deploy upgrade-runtime-deps lint
77

88
check-node-version:
99
@major="$$(node -p 'process.versions.node.split(".")[0]')"; \
@@ -92,6 +92,12 @@ score-example-criteria:
9292
check-quality-scores:
9393
$(PY) scripts/check_quality_scores.py
9494

95+
# Manual gate, not run in CI: fail while a quality waiver has 30 days or fewer
96+
# left, so an expiry can be caught before it turns every pull request red.
97+
# `make verify` (CI) already prints a warning inside the same 30-day window.
98+
check-waiver-expiry:
99+
$(PY) scripts/check_quality_scores.py --fail-within-days 30
100+
95101
check-no-figure-rationales:
96102
$(PY) scripts/check_no_figure_rationales.py
97103

@@ -121,12 +127,24 @@ dev: node-deps
121127
uv run --group workers pywrangler dev --port 9696
122128

123129
smoke-deployment:
124-
$(PY) scripts/smoke_deployment.py $(URL)
130+
$(PY) scripts/smoke_deployment.py $(URL) $(SMOKE_ARGS)
131+
132+
# Origin that `make deploy` smoke-tests once Wrangler has deployed. With
133+
# Turnstile challenges enabled, export PBE_SMOKE_BYPASS_SECRET so the POST
134+
# checks can run; SMOKE_ARGS passes extra flags (e.g. --skip-post) explicitly.
135+
DEPLOY_URL ?= https://www.pythonbyexample.dev
136+
137+
post-deploy-smoke:
138+
@$(MAKE) --no-print-directory smoke-deployment URL=$(DEPLOY_URL) || { \
139+
echo "Deployment smoke FAILED for $(DEPLOY_URL). The new version is already live: investigate now, or roll back with 'uv run --group workers pywrangler rollback'." >&2; \
140+
exit 1; \
141+
}
125142

126143
deploy: node-deps check-generated
127144
uv run --group workers pywrangler sync --force
128145
git diff --exit-code -- pylock.toml
129146
uv run --group workers pywrangler deploy
147+
@$(MAKE) --no-print-directory post-deploy-smoke
130148

131149
# Production vendors pylock.toml; tests run against uv.lock. Refresh both together.
132150
upgrade-runtime-deps: node-deps

‎README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ git diff --check
125125
- Worker Cache API keys include the HTML version
126126
- prototype layout pages are not cached
127127

128-
`make browser-layout-test` launches headless Chrome and checks the rendered Shiki code-block layout so generated line markup does not create visual blank rows.
128+
`make browser-layout-test` launches headless Chrome against the local Worker. It checks the rendered Shiki code-block layout so generated line markup does not create visual blank rows, drives the runner, sharing, copy, search and keyboard navigation through the page, and measures computed styles (pressed states, touch targets, contrast in both themes, the reader's font size, reduced-transparency and more-contrast fallbacks) in place of unit tests that matched CSS text.
129129

130130
## Asset fingerprinting and cache busting
131131

@@ -174,7 +174,7 @@ scripts/format_examples.py --check
174174
make deploy
175175
```
176176

177-
`make deploy` first runs `make check-generated`, which rebuilds and rejects any generated output not committed to the branch. It then syncs the ignored Python Workers dependency bundle from the committed `pylock.toml`, refusing to deploy if the sync would change that lock, before Wrangler deploys.
177+
`make deploy` first runs `make check-generated`, which rebuilds and rejects any generated output not committed to the branch. It then syncs the ignored Python Workers dependency bundle from the committed `pylock.toml`, refusing to deploy if the sync would change that lock, before Wrangler deploys. Finally it runs `scripts/smoke_deployment.py` against `DEPLOY_URL` (default `https://www.pythonbyexample.dev`) and exits non-zero if the deployed Worker fails any GET or POST check. Export `PBE_SMOKE_BYPASS_SECRET` when Turnstile challenges are enabled so the POST checks can run; `SMOKE_ARGS` passes extra flags such as `--skip-post` explicitly.
178178

179179
## Updating dependencies
180180

@@ -228,6 +228,10 @@ JPEGs under `public/og/` with headless Chrome (set `CHROME_PATH` if needed).
228228
footgun, notes, program/cell, prose-duplication, inline-link, scoring,
229229
figure, journey, and example-graph gates.
230230

231+
`make check-waiver-expiry` is a manual check, not run in CI: it fails while
232+
any quality waiver has 30 days or fewer left. `make quality-checks` (and so CI)
233+
only prints a warning in that window, then fails once the waiver has expired.
234+
231235
`src/example_sources_data.py` is generated and committed so Cloudflare Workers can load examples in production. Do not edit it by hand.
232236

233237
For a Python version migration, update `python_version` and `docs_base_url` in `src/example_sources/manifest.toml`, then run:

‎docs/lessons-learned.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ This document records project lessons that should guide future changes to Python
4040
- Keep the complete editable program visible because it is the thing that actually runs.
4141
- When storing examples as Markdown, keep the full editable program in `:::program` and teaching fragments in separate `:::cell` blocks. Do not concatenate cells to recreate the editor source.
4242
- Fine-grained cells can restate definitions to stay executable. This is better than collapsing class, property, recursion, match, or type-hint examples into one large cell.
43-
- Preserve a frozen golden catalog while migrating source formats. Full stdout parity is not enough; rendered teaching-cell structure must also match. (After the migration, the fixture's job changed: it is now a reviewed structural snapshot, refreshed explicitly by `scripts/refresh_golden_fixture.py`, that catches loader/parser regressions rather than guarding a migration.)
43+
- Preserve a frozen golden catalog while migrating source formats. Full stdout parity is not enough; rendered teaching-cell structure must also match. (After the migration the frozen fixture and its refresh script were retired in 0929c25, 2026-06-13; loader and parser regressions are now caught by the frontmatter property tests in `tests/test_parser_properties.py` and the byte-exact output check in `scripts/verify_examples.py`.)
4444
- Beware broad-surface titles that quietly teach only one narrow slice. Pages named `Testing`, `Packages`, `Regular Expressions`, `Type Hints`, `Async Await`, or `Special Methods` must either cover the forms a reader reasonably expects or explicitly frame themselves as a first pass and link to focused neighbors.
4545
- Do not let an important syntax form live only in a separate page if another page title strongly implies it. An umbrella `Operators` page should at least point to and lightly show `:=`, even though `Assignment Expressions` remains the focused lesson.
4646
- Journey order should follow prerequisite thinking, not catalog order. Put booleans before truthiness and conditionals, scope before closures, bytes before networking, and environment boundaries before subprocess/thread boundaries.
@@ -64,6 +64,7 @@ This document records project lessons that should guide future changes to Python
6464
- Avoid layout shifts after execution. Reserve space for metadata such as execution time before a run occurs.
6565
- When two columns use the orange rail, source and output need identical rail spacing. Differences in border/padding make the page feel broken even if the content is correct.
6666
- Use browser screenshot tests for visual bugs. Static HTML/CSS assertions are useful but can miss actual rendered layout behavior.
67+
- Assert what the browser computes, not what `site.css` says. Unit tests that matched literal CSS or JS text broke on harmless refactors and passed on real regressions, so `scripts/check_browser_layout.mjs` now measures the same intentions: computed transforms under a forced `:active`, touch-target heights, contrast in both themes, the reader's font size, emulated `prefers-reduced-transparency`/`prefers-contrast`, and runner behaviour driven through the page. To add a visual rule, add a measurement there that fails when the rule is removed.
6768

6869
## Testing and verification
6970

@@ -108,7 +109,7 @@ git diff --check
108109
- **Score what's shipping, not what was designed.** A scoring dict on the gestalt is design-time review. Production figures live in `src/marginalia.py` `FIGURES` and may have been redesigned during promotion. Scoring should track the production version with the gestalt as separate history.
109110
- **Semantic diagram review has four objects: page, cell, paint function, caption.** Geometry contracts can prove that a figure renders, but not that it still teaches the adjacent cell. On every diagram pass, read the current example cell, the attached paint function, the caption, and the score rationale together. The same audit caught `container-protocols` still showing `iter()/next()` after the page had shifted to `__setitem__`/`__contains__`/`__getitem__`, `structured-data-shapes` over-focusing on `TypedDict` while anchored to a dataclass cell, and `object-lifecycle` mentioning `__del__` after the lesson had been reframed around references.
110111
- **Some examples should never have figures.** Constraint-shaped, infrastructure-shaped, and aggregator-shaped slugs lack a single mechanism to depict. Force-fitting figures on them scores below the gate. Leave them figure-less and document why rather than ship weak figures.
111-
- **Audits without contracts rot; bug classes need automated gates.** When you find a bug class — clipping, collision, off-palette colour, drifting twin coordinates, duplicate caption — write a unit test that asserts the invariant across every figure. The geometry contracts in `tests/test_marginalia_geometry.py` started as ad-hoc scripts and were promoted to CI gates after the same bug class recurred. 54 tests today cover 9 contract families; new figures pass them automatically because each test iterates `FIGURES`.
112+
- **Audits without contracts rot; bug classes need automated gates.** When you find a bug class — clipping, collision, off-palette colour, drifting twin coordinates, duplicate caption — write a unit test that asserts the invariant across every figure. The geometry contracts in `tests/test_marginalia_geometry.py` started as ad-hoc scripts and were promoted to CI gates after the same bug class recurred. Each contract family is a test class there; new figures pass them automatically because each test iterates `FIGURES`. (Count the classes rather than quoting a number here: an earlier "54 tests, 9 families" claim drifted from the file.)
112113
- **Clipping ≠ collision; padding the viewBox fixes one but not the other.** The `value-types` bug had two components: the first `INT` tag was clipped above the viewBox (geometry escapes its frame), and the `STR`/`LIST`/`DICT` tags overlapped the boxes above them (geometry collides internally). Padding the viewBox solved (1) and disguised (2). Element-element collision needs its own audit that walks every text-rect, text-text, and (for label-on-edge cases) text-line bounding-box pair.
113114
- **Heuristic audits over-flag; trust the design, not the regex.** Probes for "prose duplication" (SVG text matching caption substring), "text crossing a line" (label bbox bisected by a hairline), and "text overlapping a circle" each surfaced ~3-10 hits — all false positives. Diagrammatic labels naturally appear in captions (`__getattr__`, `yield from inner`); dashed strikes through `.append` are deliberate; text inside node circles is the design. Don't promote heuristic findings to contracts without confirming each is a real bug.
114115
- **Structural twins must share coordinates exactly.** When two figures depict parallel concepts — `kw-only-separator` and `positional-only-separator`, `class-triangle` and `metaclass-triangle` — they read as a pair. A single-pixel drift in one breaks the visual rhyme. Treat a coordinate change in one as a forced change in both, in the same commit. The audit caught `kw-only-separator` at the old `x=82` after `positional-only-separator` had moved to the corrected `x=75`.

0 commit comments

Comments
 (0)