feat: publish and gate the Latin-shape exposure set (#815) - #930
Closed
raeq wants to merge 4 commits into
Closed
Conversation
|
📄 Docs preview: https://b7e9d5bf.disarm-docs.pages.dev |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new tests and generator currently repeat a full Unicode census multiple times, which will significantly slow CI and local runs unless cached/reused.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Publishes the “Latin-shape exposure set” requested in #815 as a reviewed TSV fixture and adds gates to prevent the set from drifting (including per-block counts), documenting the limitation that the census only measures unfolded code points.
Changes:
- Add
scripts/gen_latin_shape_exposure.pyto generate the exposure-set fixture and support--checkgating. - Add
tests/test_latin_shape_exposure.pyto validate the fixture against live library behavior and gate per-block counts plus key exclusions. - Update documentation and release notes to reflect the new published/gated exposure set and its limitations.
File summaries
| File | Description |
|---|---|
| tests/test_latin_shape_exposure.py | Adds a test gate to keep the exposure fixture aligned with the generator/library and asserts key selector invariants. |
| tests/fixtures/latin_shape_exposure.tsv | Introduces the generated exposure-set fixture (299 rows) used as the reviewable, non-drifting published list. |
| scripts/gen_latin_shape_exposure.py | Implements the census + selector used to generate/check the exposure fixture. |
| docs/limitations.md | Updates the limitations narrative to point at the published fixture and record the “wrong-fold” caveat (#916). |
| CHANGELOG.md | Adds an Unreleased entry documenting the new exposure-set publication and gate. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Items 4, 5 and 7. 299 non-ASCII code points that read as a Latin letter reach ASCII on no surface, published as `tests/fixtures/latin_shape_exposure.tsv` with a per-block gate. Per block and not only in total, deliberately: a table refresh that gained rows in one block and lost them in another would pass a total-only check while changing what a deployment is exposed to. An exposure set rather than a bug list. Latin Extended-D contributes 146 of the 299 and they are medievalist letters with no sensible ASCII fold; the useful output is a reviewed list, which is what #815 asked for. The selector is the part worth reading, because the issue's number was wrong in both directions and my first correction was wrong three more ways. #815 matched names STARTING with `LATIN `, `MODIFIER LETTER ` or `TURNED `, or containing `SMALL CAPITAL`, so `NEGATIVE CIRCLED LATIN CAPITAL LETTER A` and 51 siblings sat outside the count. Widening it to "the name contains LATIN" then pulled in: - 53 COMBINING marks — diacritics over a base, not letters standing in for one, and `strip_accents`' business; - 52 TAG characters — stripped rather than folded (#413), which reads as unhandled to a test that only asks whether ASCII came out; - `LATIN CROSS`, a symbol, and `GLAGOLITIC CAPITAL LETTER LATINATE MYSLITE`, which is Glagolitic. Every one of those three I found by reading the generator's output, not by reasoning about the pattern. The selector is now a word-bounded `LATIN [CAPITAL|SMALL] LETTER` plus a category of letter or symbol, which excludes marks and format characters by construction. One caveat recorded beside the number, because the census cannot express it: it counts UNFOLDED code points, so one that folds to the WRONG letter is invisible to it. That is #916, and it is why the 299 is a floor rather than a measure of correctness. Tests assert the fixture matches the library, each block holds its count, the three exclusion classes stay excluded, what #915 and #920 fixed is absent, and the selector is pointed at something. Refs #815, #916, #413, #915, #920. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
`gen.census()` walks the codespace and calls six disarm surfaces per selected code point. It ran once for the fixture comparison and again for every parametrized block — nine sweeps where one would do. A module-scoped fixture now runs it once: 1.20s -> 0.22s. Fourth time this shape has come up in this cycle — #920's `_enclosed()`, #923's reverse sweep, #929's `_small_caps()`, and now this — so the fixture's docstring names the pattern rather than being quietly corrected again: a helper that reads as cheap at the call site and walks the codespace inside. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
`main()` called `census()` for `render()` and again for the row count in the status line, so rewriting the fixture walked the codespace twice. Computed once. Fifth instance of this shape in the cycle and the first in non-test code, which is the part worth noting: I had been treating it as a testing habit. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
CI failed on `Latin Extended-D: 143 now, 146 in the fixture`. Not a coverage change —
three different UCDs are in play and the fixture was generated against one of them:
disarm's bundled tables 17.0.0
this checkout's host 16.0.0
CI's Python 3.12 15.1.0
The selector reads `unicodedata.name`, so a code point assigned in 16.0 is invisible to an
older interpreter and the totals cannot match. disarm exposes no name or category API, so
the census has to use the host's, which makes a fixed per-code-point fixture
interpreter-dependent by construction.
The gate now compares only the code points the RUNNING interpreter can name. A host with
an older UCD sees a subset and still agrees, while the thing the gate exists for still
fails: a code point the host CAN name changing status. Verified by hiding Latin Extended-G
and watching the comparison drop from 299 to 262 rows without failing.
Also removed `tests/test_adversarial_corpora.py` from this branch. It is #732's work and
was swept in by a `git add -A` while it sat untracked in the tree — which is how it came
to reference a page that does not exist on this branch, and is the second failure in this
CI run.
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Owner
Author
|
Superseded by a clean rebuild off Rebuilt as a single commit with the same content plus the UCD fix CI found here. See the replacement PR. |
raeq
added a commit
that referenced
this pull request
Sep 3, 2026
Three comments cited "#930 review" as their reason. That PR is closed — it was replaced by this one — so a reader following the reference lands on a superseded branch rather than an explanation. The reasons are inline now: the census is computed once because it walks the codespace and calls six surfaces per selected code point; the fixture is module-scoped for the same reason; the gate compares what the interpreter can name because a total comparison fails on a Python 3.12 runner for reasons unrelated to coverage. A list of PR numbers illustrating the repeated-sweep pattern went the same way. The pattern is worth keeping and the numbers age badly. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq
added a commit
that referenced
this pull request
Sep 3, 2026
* feat: publish and gate the Latin-shape exposure set (#815) Items 4, 5 and 7. 299 non-ASCII code points that read as a Latin letter reach ASCII on no surface, published as `tests/fixtures/latin_shape_exposure.tsv` with a per-block gate — per block and not only in total, because a refresh that gained rows in one block and lost them in another would pass a total-only check while changing what a deployment is exposed to. An exposure set rather than a bug list. Latin Extended-D contributes 146 of the 299 and they are medievalist letters with no sensible ASCII fold. The selector is item 7 and the part worth reading. #815 matched names STARTING with `LATIN `, `MODIFIER LETTER ` or `TURNED `, or containing `SMALL CAPITAL`, so `NEGATIVE CIRCLED LATIN CAPITAL LETTER A` and 51 siblings were never inside the number. Widening it to "the name contains LATIN" then pulled in three classes that are not exposure, each found by reading the output rather than reasoning about the pattern: - 53 COMBINING marks, which are diacritics over a base and `strip_accents`' business; - 52 TAG characters, which are stripped rather than folded (#413), so correct handling reads as a gap to a test that only asks whether ASCII came out; - `LATIN CROSS`, a symbol, and `GLAGOLITIC CAPITAL LETTER LATINATE MYSLITE`. The gate compares only the code points the RUNNING interpreter can name. Three UCDs are in play — disarm's tables at 17.0.0, this host at 16.0.0, CI's Python 3.12 at 15.1.0 — and the selector reads `unicodedata.name`, so a fixed per-code-point fixture is interpreter- dependent by construction. Comparing on what the host can see is deterministic anywhere and still fails on what the gate is for. Verified by hiding Latin Extended-G and watching the comparison drop 299 -> 262 without failing. One caveat recorded beside the number: the census counts UNFOLDED code points, so one that folds to the WRONG letter is invisible to it. That is #916, and it is why 299 is a floor. Rebuilt as a single commit off main. The previous branch carried #732's test file, swept in by a `git add -A` while it sat untracked, and rebasing after #732 merged conflicted between adding and deleting a file main now owns. Refs #815, #916, #413, #915, #920. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com> * docs: comments pointed at a superseded PR (#932 review) Three comments cited "#930 review" as their reason. That PR is closed — it was replaced by this one — so a reader following the reference lands on a superseded branch rather than an explanation. The reasons are inline now: the census is computed once because it walks the codespace and calls six surfaces per selected code point; the fixture is module-scoped for the same reason; the gate compares what the interpreter can name because a total comparison fails on a Python 3.12 runner for reasons unrelated to coverage. A list of PR numbers illustrating the repeated-sweep pattern went the same way. The pattern is worth keeping and the numbers age badly. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Richard Quinn <quinn.richard@gmail.com> --------- Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#815 items 4, 5 and 7. Stacked behind #929 (item 3), which is independent of this.
What lands
tests/fixtures/latin_shape_exposure.tsv— 299 non-ASCII code points that read as a Latin letter and reach ASCII on no surface, with a per-block gate.Per block and not only in total, deliberately: a refresh that gained rows in one block and lost them in another would pass a total-only check while changing what a deployment is exposed to.
An exposure set, not a bug list: Latin Extended-D is 146 of the 299 and they are medievalist letters with no sensible ASCII fold. The useful output is a reviewed list, which is what the issue asked for.
The selector, which is item 7 and the interesting part
The issue's 401 was wrong in both directions, and my first correction was wrong three more ways.
#815 matched names starting with
LATIN,MODIFIER LETTERorTURNED, or containingSMALL CAPITAL— soNEGATIVE CIRCLED LATIN CAPITAL LETTER Aand 51 siblings were never inside the number. Widening to "the name contains LATIN" then pulled in:COMBINING LATIN SMALL LETTER …strip_accents' businessTAG LATIN … LETTER …LATIN CROSS,LATINATE MYSLITEI found all three by reading the generator's output, not by reasoning about the pattern. The selector is now a word-bounded
LATIN [CAPITAL|SMALL] LETTERplus a category of letter or symbol, which excludes marks and format characters by construction.The caveat the census cannot express
It counts unfolded code points, so one that folds to the wrong letter is invisible to it. That is #916, and it is why 299 is a floor rather than a measure of correctness. Recorded on
limitations.mdbeside the number.Tests
The fixture matches the library; each block holds its count; the three exclusion classes stay excluded; what #915 and #920 fixed is absent from the set; and the selector is pointed at something.
Gate
cargo fmt/clippy·cargo test·ruff check/format .·mypy·mkdocs --strict· 41/41 doc pages · 6725 passed, 38 skipped.🤖 Generated with Claude Code