Skip to content

feat: the 54 negative enclosed letters fold to their letter (#815) - #920

Merged
raeq merged 2 commits into
mainfrom
fix/815-enclosed-letters
Sep 2, 2026
Merged

feat: the 54 negative enclosed letters fold to their letter (#815)#920
raeq merged 2 commits into
mainfrom
fix/815-enclosed-letters

Conversation

@raeq

@raeq raeq commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Refs #815, scope item 8 — the gap the styles census turned up.

The asymmetry

🅐 U+1F150 and 🅰 U+1F170 folded on no surface. Their positive counterparts and 🄰 do, because NFKC decomposes those and leaves these alone. Two neighbouring blocks, opposite outcomes, and a "fancy text" generator offers both side by side — so one style was neutralised and the other passed through untouched.

Derived from the UCD name and filtered to what NFKC does not already handle, landing on exactly the 54 that need it: NEGATIVE CIRCLED (26), NEGATIVE SQUARED (26), CROSSED NEGATIVE SQUARED, and one stray SQUARED.

Two families match the same name pattern and are excluded, with tests saying so rather than leaving it implicit:

excluded why
Tags (52) stripped as a smuggling class (#413); canonicalize already returns ab and has_anomalies fires
Combining letters (23) category Mn — a diacritic over a base, strip_accents' business

Three things worth reading the diff for

The build.rs overlap gate fired, which is what it exists for. Four of the 54 are also CLDR emoji — 🅰 is NEGATIVE SQUARED LATIN CAPITAL LETTER A and the blood-type A button. They join the set #614 reviewed; count 50 → 54. canonicalize folds them per #614's precedence (inside a comparison preset the fold wins, or a spoof and its target stop being equal). llm_guardrail still names them, because demojize runs before the fold there — that divergence is #918's subject, not introduced here.

The first pass folded every capital to lowercase. 🅰 reached a while 🄰 reaches A. These are category So, so fix_case_mismatch can't tell a capital from a small letter. Case now comes from the UCD name, and a test asserts a capital never folds to lowercase — reintroducing an asymmetry is a poor way to remove one.

The name pattern matched 319 before narrowing: 52 plain ASCII letters (.* matching empty) plus most of the enclosed blocks, which NFKC already handles.

⚠️ Key schema 4 → 5

catalog_key moves for all 54; search_key and sort_key do not.

The corpus contained none of these — the third time this cycle the fixture stayed green through a key change because it did not sample the class being fixed. 58 rows added.

The census does not move, and that is the point

It stays 401. #815's sweep selects on names starting with LATIN /MODIFIER LETTER /TURNED or containing SMALL CAPITAL, and NEGATIVE CIRCLED LATIN CAPITAL LETTER A matches none. The 54 were never inside the number. A selector matching LATIN … LETTER anywhere counts 469 of 1817 — which is what the issue's scope item 7 is about.

Gate

cargo fmt/clippy · cargo test · cargo doc · perf_lint.sh · ruff · mypy · mkdocs --strict · audit_language_consistency · 6615 passed, 31 skipped.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 2, 2026 19:14
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

📄 Docs preview: https://d6cd8fad.disarm-docs.pages.dev

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new test module repeatedly rescans the full Unicode codespace without caching, which can unnecessarily slow the test suite and should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds TR39/ASCII-fold coverage for the 54 “negative enclosed” Latin letter symbols (e.g., 🅐/🅰), closing an asymmetry where positive enclosed forms folded via NFKC but their negative counterparts did not. This updates generated confusables data, adjusts the emoji/confusable overlap gate, bumps key schema (catalog_key changes), and extends tests/docs accordingly.

Changes:

  • Generate + ship 54 new confusables_to_latin mappings for NEGATIVE CIRCLED/SQUARED (plus crossed negative squared + one squared) enclosed letters.
  • Add dedicated regression tests (including dual-purpose emoji buttons) and update the key-stability corpus + schema/version metadata.
  • Refresh counts and narrative across docs, Python docstrings, changelog, and the build-time overlap gate (50 → 54).
File summaries
File Description
tests/test_non_emoji_names.py Updates #614 overlap characterization expectations (50 → 54) and documents why.
tests/test_enclosed_letter_folds.py New regression suite for the 54 negative enclosed letter folds + exclusions + corpus coverage.
tests/fixtures/key_stability/README.md Documents corpus growth due to added negative-enclosed rows.
tests/fixtures/key_stability/corpus.txt Adds 58 rows sampling the newly-folded class (and mixed examples).
src/tables/data/confusables_to_latin.tsv Adds 54 new enclosed-letter fold rows.
src/tables/confusables_data.rs Updates the documented Latin mapping count.
src/api/metadata.rs Bumps KEY_SCHEMA_VERSION (4 → 5) and updates fixture SHA.
scripts/gen_confusables.py Adds name-derived enclosed-letter folds to generation pipeline and merges into ASCII_FOLD.
python/disarm/_api.py Updates documented Latin mapping count in normalize_confusables docstring.
docs/user-guide/confusables.md Updates confusables table counts for Latin target.
docs/limitations.md Updates bundled-row count in limitations discussion.
docs/architecture/data-tables.md Updates confusables table entry count.
CHANGELOG.md Adds release note + upgrade note for key schema bump.
build.rs Updates emoji/confusable overlap gate expected size (50 → 54) and message.
Review details

Suppressed comments (2)

tests/test_enclosed_letter_folds.py:33

  • _enclosed() is recomputed on every call, but the result is deterministic for a given interpreter/UCD version. Caching it (and compiling the regex once) avoids repeating the full codespace scan across tests in this file.

def _enclosed() -> dict[str, str]:
    """Every code point the generator's rule selects, re-derived here independently."""
    import re

tests/test_enclosed_letter_folds.py:40

  • After moving the pattern to module scope, use the compiled regex here to avoid recompiling it on every iteration of the sweep.
        ch = chr(cp)
        m = re.fullmatch(r".+\bLATIN (CAPITAL|SMALL) LETTER ([A-Z])", unicodedata.name(ch, ""))
        if not m or ch.isascii():
  • Files reviewed: 14/15 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_enclosed_letter_folds.py
`🅐` U+1F150 and `🅰` U+1F170 folded on no surface. Their positive counterparts `ⓐ` and
`🄰` do, because NFKC decomposes those and leaves these alone. Two neighbouring blocks,
opposite outcomes, and a "fancy text" generator offers both side by side — so one style
was neutralised and the other passed through untouched.

Derived from the UCD name, filtered to what NFKC does not already handle, which lands on
exactly the 54 that need it. Two families matching the same name pattern are excluded
because both are already handled correctly, and the tests say so rather than leaving it
implicit: Tags are stripped as a smuggling class (#413), and combining letters are
category `Mn` and belong to `strip_accents`.

Three things this turned up that are worth reading the diff for:

- The `build.rs` overlap gate fired. Four of the 54 are also CLDR emoji — the blood-type
  and parking buttons — so they join the set #614 reviewed. Count 50 -> 54. `canonicalize`
  folds them, per #614's precedence: inside a comparison preset the fold wins over the
  name, or a spoof and its target stop being equal. `llm_guardrail` still names them
  because demojize runs before the fold there, which is #918.
- The first pass folded every capital to a LOWERCASE letter, so `🅰` reached `a` while its
  positive counterpart `🄰` reaches `A`. These are category `So`, so `fix_case_mismatch`
  cannot tell a capital from a small letter. Case now comes from the UCD name, and a test
  asserts a capital never folds to lowercase — the asymmetry this set exists to remove is
  not one to reintroduce.
- The name pattern matched 319 code points before it was narrowed: 52 plain ASCII letters
  (`.*` matching empty) and most of the enclosed blocks, which NFKC already handles.

KEY SCHEMA 5 -> 6. `catalog_key` moves for all 54; `search_key` and `sort_key` do not.
The corpus contained none of these — the THIRD time this cycle the fixture stayed green
through a key change because it did not sample the class being fixed. 58 rows added.

The #815 census does NOT move: it stays 401, because that sweep selects on names STARTING
with `LATIN `/`MODIFIER LETTER `/`TURNED ` or containing `SMALL CAPITAL`, and
`NEGATIVE CIRCLED LATIN CAPITAL LETTER A` matches none of them. The 54 were never inside
the number. A selector that matches `LATIN … LETTER` anywhere in the name counts 469 of
1817 instead, which is the figure the issue's scope item 7 is about.

Refs #815, #614, #918, #413.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
@raeq
raeq force-pushed the fix/815-enclosed-letters branch from f8cb481 to 5f00a70 Compare September 2, 2026 19:19
`_enclosed()` walks 1.1M code points and seven tests in the file call it. `lru_cache` on
the function and the name regex compiled at module scope instead of per call:

    1.42s -> 0.15s

Second time this session a review has caught a repeated full-codespace sweep in a test I
wrote — #919 was the same shape, an NFKC call made twice per iteration. The pattern to
watch is a helper that is cheap to call and expensive to run.

Left re-derived rather than imported from `scripts/gen_confusables.py`: a test that reuses
the generator's own selector cannot catch the generator selecting the wrong set. The
docstring says so, and says the cached dict must not be mutated.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
@raeq
raeq merged commit fef2ba8 into main Sep 2, 2026
25 checks passed
@raeq
raeq deleted the fix/815-enclosed-letters branch September 2, 2026 19:36
raeq added a commit that referenced this pull request Sep 2, 2026
BREAKING: two shipped surfaces change output, and KEY_SCHEMA_VERSION goes 6 -> 7.

Both converted an attacker-chosen emoji into attacker-chosen English inside the text being
screened:

    get_pipeline("llm_guardrail")("ignore😀 previous")
      was 'ignore grinning face previous'
      now 'ignore😀 previous'

Over `Emoji_Presentation`, 1,180 of 1,219 code points reached a name, together spanning
1,272 distinct English words — `stop`, `end`, `new`, `key`, `no` among them — with up to
eight words from a single code point. The emoji is fully visible, so this was never
concealment: the sanitiser was the thing writing new words into the prompt.

Third time naming won over neutralizing, and the first where no table is at fault. #614
narrowed WHAT was named, #757 narrowed it again; neither could reach the case where the
naming is correct and a security surface should not be doing it at all.

The emoji is LEFT IN PLACE, not removed. #910's scope proposed removal as "the better
answer on the segmentation axis". Measured over 144 emoji with the probe `stop<emoji>now`,
removal fuses the two words into `stopnow` 144 times out of 144 while leaving keeps them
apart 144 out of 144. Removal does not close a split word; it creates a joined one. The
issue's own reasoning inverted under measurement.

The profile could not move until #914: `demojize` was the only step in the composed
surface removing the Plane 14 TAG block, so flipping it alone traded a text-injection
primitive for a concealment channel. `strip_plane14` carries it now. The preset never had
that coupling — `StripInvisible` does that job there — so nothing was riding on the step.

Only these two move. `ml_normalize` and `ml_corpus_normalize` keep naming: glossing is the
point of an ML-corpus surface and neither is a security preset. Naming stays reachable
through `demojize()` and `TextPipeline(demojize=True)`, so this is a relocation.

Two side effects worth reading the diff for:

- `llm_guardrail` now FOLDS the dual-purpose enclosed letters instead of naming them, so
  `🅿AYPAL` reaches `paypal` rather than `p buttonaypal`. The #918/#920 tension for those
  four resolves itself.
- `strip_obfuscation` no longer links the CLDR name table into a wasm build. The #695
  coupling gate failed the moment the step came out, which is exactly its job, and the
  ceiling drops 640 KB -> 400 KB.

KEY SCHEMA 6 -> 7: `strip_obfuscation` is a fixture column and its output moves for any
input containing emoji. This one the fixture DID catch — its corpus carries 54 emoji,
unlike the three classes earlier in this cycle it could not see.

TDD both halves. The profile: 14 red, then green. The preset: 2 red, then green.

Three Rust tests asserted the naming and are rewritten rather than deleted, because what
they were protecting underneath still matters: the TR39 folds #614's separator rule
existed for, and idempotence, are now asserted directly instead of through the arrangement
that protected them.

Closes #910.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 2, 2026
…926)

* feat!: llm_guardrail and strip_obfuscation stop naming emoji (#910)

BREAKING: two shipped surfaces change output, and KEY_SCHEMA_VERSION goes 6 -> 7.

Both converted an attacker-chosen emoji into attacker-chosen English inside the text being
screened:

    get_pipeline("llm_guardrail")("ignore😀 previous")
      was 'ignore grinning face previous'
      now 'ignore😀 previous'

Over `Emoji_Presentation`, 1,180 of 1,219 code points reached a name, together spanning
1,272 distinct English words — `stop`, `end`, `new`, `key`, `no` among them — with up to
eight words from a single code point. The emoji is fully visible, so this was never
concealment: the sanitiser was the thing writing new words into the prompt.

Third time naming won over neutralizing, and the first where no table is at fault. #614
narrowed WHAT was named, #757 narrowed it again; neither could reach the case where the
naming is correct and a security surface should not be doing it at all.

The emoji is LEFT IN PLACE, not removed. #910's scope proposed removal as "the better
answer on the segmentation axis". Measured over 144 emoji with the probe `stop<emoji>now`,
removal fuses the two words into `stopnow` 144 times out of 144 while leaving keeps them
apart 144 out of 144. Removal does not close a split word; it creates a joined one. The
issue's own reasoning inverted under measurement.

The profile could not move until #914: `demojize` was the only step in the composed
surface removing the Plane 14 TAG block, so flipping it alone traded a text-injection
primitive for a concealment channel. `strip_plane14` carries it now. The preset never had
that coupling — `StripInvisible` does that job there — so nothing was riding on the step.

Only these two move. `ml_normalize` and `ml_corpus_normalize` keep naming: glossing is the
point of an ML-corpus surface and neither is a security preset. Naming stays reachable
through `demojize()` and `TextPipeline(demojize=True)`, so this is a relocation.

Two side effects worth reading the diff for:

- `llm_guardrail` now FOLDS the dual-purpose enclosed letters instead of naming them, so
  `🅿AYPAL` reaches `paypal` rather than `p buttonaypal`. The #918/#920 tension for those
  four resolves itself.
- `strip_obfuscation` no longer links the CLDR name table into a wasm build. The #695
  coupling gate failed the moment the step came out, which is exactly its job, and the
  ceiling drops 640 KB -> 400 KB.

KEY SCHEMA 6 -> 7: `strip_obfuscation` is a fixture column and its output moves for any
input containing emoji. This one the fixture DID catch — its corpus carries 54 emoji,
unlike the three classes earlier in this cycle it could not see.

TDD both halves. The profile: 14 red, then green. The preset: 2 red, then green.

Three Rust tests asserted the naming and are rewritten rather than deleted, because what
they were protecting underneath still matters: the TR39 folds #614's separator rule
existed for, and idempotence, are now asserted directly instead of through the arrangement
that protected them.

Closes #910.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

* test: the "no new token" check was a substring search (#926 review)

The property is stated in tokens and was tested with `token not in probe.lower()`, which
passes anything that happens to sit inside an input word. Over `ignore😀 previous` that
accepts `nor`, `revi` and `ous` — so a gloss producing one of them would have read as no
new token at all, and the test would have said the profile was clean.

Compares against the input's token set now, with the tokenizer factored out so both sides
are split the same way.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

* style: ruff formats python blocks inside markdown too (#926 CI)

`Lint & format` failed on `docs/user-guide/llm-pipelines.md`. Ruff formats fenced `python`
blocks in markdown, and the comment alignment I used in the new demojize example is not
its style.

The cause was my gate, not the change: I ran `ruff format --check` over
`python/disarm tests/ scripts/` while CI runs it over `.`. Same ruff version — 0.16.5,
pinned — so a narrower path list was the whole difference.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

---------

Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 3, 2026
… the code (#929 review)

`_small_caps()` rebuilt its lookup on every call — 1.1M `unicodedata.name()` calls, once
per parametrized word. Built once now: the file goes 0.65s -> 0.09s.

Third time this has come up (#920, #923, now here), so the helper says so rather than just
being fixed: a function that reads as cheap at the call site and walks the codespace
inside is the shape to watch.

And the doc comment still said `Москва` "passes the first three and fails the fourth". It
did when there were four conditions; removing the redundant one renumbered them and the
prose did not follow. It passes 1 and 2 and fails 3, and the comment now says which
condition does the work and why — that being the whole difference between a Russian word
and Latin letters in disguise.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 3, 2026
…815) (#929)

* fix: the detector went silent exactly when the disguise was complete (#815)

`inspect_anomalies` reported a `confusable` only when a word ALSO carried an ASCII letter.
That gate is #633's and is worth having — it is what keeps `Привет` clean, and #907 is the
standing record of what happens when a surface treats legitimate non-Latin text as
something to rewrite.

It inverted. Converting one letter of `instructions` to its small capital tripped the
detector; converting all twelve silenced it, because the last ASCII letter left with the
last substitution. A word where every character imitates a Latin letter is more suspicious
than a half-converted one, not less.

A token is reported now when every character folds to an ASCII letter, there are at least
four of them, and its script is Latin or none. The script condition is the whole
separation between the two cases: every letter of `Москва` folds to ASCII too, and it is a
Russian word rather than Latin letters wearing a disguise. The "no script" arm covers the
negative enclosed letters, which are category `So`, belong to no script, and still spell a
word.

This narrows #722's spared-block list and says so. Three of the seven stay spared, because
a whole token written in them is ordinary text nothing else spells: Halfwidth and
Fullwidth Forms, CJK Compatibility, Letterlike Symbols. `NHK` does not fire and neither
does `paypal` — #633's trade, unchanged. Phonetic Extensions and Enclosed Alphanumeric
Supplement are not spared, because those are the blocks that spell `ᴘᴀꜱꜱᴡᴏʀᴅ`. Sparing a
block because it CAN hold ordinary text is what let the finished attack through.

Validated before implementing, not after: 0 hits across 235,976 entries of
`/usr/share/dict/words`, and 4 across the 23,135-row key-stability corpus — every one an
attack string added by #915.

TDD, and it corrected me twice. The first draft asserted "IPA stays clean" for
transcriptions that already fire today, which is #633's gate working as designed on mixed
spelling; measured and split into what the rule governs and what predates it.

Mutation testing then found the rule had a redundant condition. Deleting the "contains no
ASCII letter" check left all 89 tests passing, because the table has three ASCII sources
(#725) and none is a letter, so the fold test already rejects one. Removed rather than
left in as a guard that reads load-bearing and is not, and the Rust unit test says why.

Refs #815, #633, #722, #907, #915, #920.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

* test: cache the codespace sweep, and the doc numbering did not follow the code (#929 review)

`_small_caps()` rebuilt its lookup on every call — 1.1M `unicodedata.name()` calls, once
per parametrized word. Built once now: the file goes 0.65s -> 0.09s.

Third time this has come up (#920, #923, now here), so the helper says so rather than just
being fixed: a function that reads as cheap at the call site and walks the codespace
inside is the shape to watch.

And the doc comment still said `Москва` "passes the first three and fails the fourth". It
did when there were four conditions; removing the redundant one renumbered them and the
prose did not follow. It passes 1 and 2 and fails 3, and the comment now says which
condition does the work and why — that being the whole difference between a Russian word
and Latin letters in disguise.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

---------

Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 3, 2026
Items 4, 5 and 7. 299 non-ASCII code points that read as a Latin letter reach ASCII on no
surface, published as `tests/fixtures/latin_shape_exposure.tsv` with a per-block gate.

Per block and not only in total, deliberately: a table refresh that gained rows in one
block and lost them in another would pass a total-only check while changing what a
deployment is exposed to.

An exposure set rather than a bug list. Latin Extended-D contributes 146 of the 299 and
they are medievalist letters with no sensible ASCII fold; the useful output is a reviewed
list, which is what #815 asked for.

The selector is the part worth reading, because the issue's number was wrong in both
directions and my first correction was wrong three more ways.

#815 matched names STARTING with `LATIN `, `MODIFIER LETTER ` or `TURNED `, or containing
`SMALL CAPITAL`, so `NEGATIVE CIRCLED LATIN CAPITAL LETTER A` and 51 siblings sat outside
the count. Widening it to "the name contains LATIN" then pulled in:

- 53 COMBINING marks — diacritics over a base, not letters standing in for one, and
  `strip_accents`' business;
- 52 TAG characters — stripped rather than folded (#413), which reads as unhandled to a
  test that only asks whether ASCII came out;
- `LATIN CROSS`, a symbol, and `GLAGOLITIC CAPITAL LETTER LATINATE MYSLITE`, which is
  Glagolitic.

Every one of those three I found by reading the generator's output, not by reasoning about
the pattern. The selector is now a word-bounded `LATIN [CAPITAL|SMALL] LETTER` plus a
category of letter or symbol, which excludes marks and format characters by construction.

One caveat recorded beside the number, because the census cannot express it: it counts
UNFOLDED code points, so one that folds to the WRONG letter is invisible to it. That is
#916, and it is why the 299 is a floor rather than a measure of correctness.

Tests assert the fixture matches the library, each block holds its count, the three
exclusion classes stay excluded, what #915 and #920 fixed is absent, and the selector is
pointed at something.

Refs #815, #916, #413, #915, #920.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 3, 2026
`gen.census()` walks the codespace and calls six disarm surfaces per selected code point.
It ran once for the fixture comparison and again for every parametrized block — nine
sweeps where one would do. A module-scoped fixture now runs it once: 1.20s -> 0.22s.

Fourth time this shape has come up in this cycle — #920's `_enclosed()`, #923's reverse
sweep, #929's `_small_caps()`, and now this — so the fixture's docstring names the pattern
rather than being quietly corrected again: a helper that reads as cheap at the call site
and walks the codespace inside.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 3, 2026
* feat: publish and gate the Latin-shape exposure set (#815)

Items 4, 5 and 7. 299 non-ASCII code points that read as a Latin letter reach ASCII on no
surface, published as `tests/fixtures/latin_shape_exposure.tsv` with a per-block gate —
per block and not only in total, because a refresh that gained rows in one block and lost
them in another would pass a total-only check while changing what a deployment is exposed
to.

An exposure set rather than a bug list. Latin Extended-D contributes 146 of the 299 and
they are medievalist letters with no sensible ASCII fold.

The selector is item 7 and the part worth reading. #815 matched names STARTING with
`LATIN `, `MODIFIER LETTER ` or `TURNED `, or containing `SMALL CAPITAL`, so
`NEGATIVE CIRCLED LATIN CAPITAL LETTER A` and 51 siblings were never inside the number.
Widening it to "the name contains LATIN" then pulled in three classes that are not
exposure, each found by reading the output rather than reasoning about the pattern:

- 53 COMBINING marks, which are diacritics over a base and `strip_accents`' business;
- 52 TAG characters, which are stripped rather than folded (#413), so correct handling
  reads as a gap to a test that only asks whether ASCII came out;
- `LATIN CROSS`, a symbol, and `GLAGOLITIC CAPITAL LETTER LATINATE MYSLITE`.

The gate compares only the code points the RUNNING interpreter can name. Three UCDs are in
play — disarm's tables at 17.0.0, this host at 16.0.0, CI's Python 3.12 at 15.1.0 — and the
selector reads `unicodedata.name`, so a fixed per-code-point fixture is interpreter-
dependent by construction. Comparing on what the host can see is deterministic anywhere
and still fails on what the gate is for. Verified by hiding Latin Extended-G and watching
the comparison drop 299 -> 262 without failing.

One caveat recorded beside the number: the census counts UNFOLDED code points, so one that
folds to the WRONG letter is invisible to it. That is #916, and it is why 299 is a floor.

Rebuilt as a single commit off main. The previous branch carried #732's test file, swept
in by a `git add -A` while it sat untracked, and rebasing after #732 merged conflicted
between adding and deleting a file main now owns.

Refs #815, #916, #413, #915, #920.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

* docs: comments pointed at a superseded PR (#932 review)

Three comments cited "#930 review" as their reason. That PR is closed — it was replaced by
this one — so a reader following the reference lands on a superseded branch rather than an
explanation.

The reasons are inline now: the census is computed once because it walks the codespace and
calls six surfaces per selected code point; the fixture is module-scoped for the same
reason; the gate compares what the interpreter can name because a total comparison fails on
a Python 3.12 runner for reasons unrelated to coverage.

A list of PR numbers illustrating the repeated-sweep pattern went the same way. The pattern
is worth keeping and the numbers age badly.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>

---------

Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq added a commit that referenced this pull request Sep 6, 2026
Ten version fields across nine files, plus the four `disarm_core` glue pins,
which a MINOR moves in the release PR itself (RELEASING.md):

  Cargo.toml, pyproject.toml, CITATION.cff
  bindings/node/package.json + package-lock.json (two fields)
  bindings/ruby/lib/disarm/version.rb
  bindings/java/{disarm-java,disarm-kotlin}/build.gradle.kts
  uv.lock (regenerated online; the only other movement is a
    `python_full_version < '3.11'` marker on typing-extensions, the lock
    catching up to pyproject rather than dependency churn. An offline
    regeneration also downgraded hypothesis from the local cache and was
    discarded.)
  bindings/{node,ruby/ext/disarm,java/rust,cabi}/Cargo.toml  0.15 -> 0.16

The two JVM documentation versions the eight numbered items do not cover:
docs/java/getting-started.md (the Kotlin DSL, Groovy and Maven XML forms) and
bindings/java/README.md.

CHANGELOG.md is stamped `[0.16.0] — 2026-09-06` under an empty `[Unreleased]`
and gains a consolidated `### Upgrade notes` section, as 0.15.0 had one.
`KEY_SCHEMA_VERSION` went 3 -> 9 in six steps this cycle (#915, #919, #920,
#926, #941, #955), each with an inline note under Added or Changed (breaking),
and nothing listed them together. One inline arrow was wrong and is corrected:
the negative-enclosed-letters entry (#920) said 4 -> 5, and the constant went
5 -> 6 at that commit. docs/RUST_API.md's upgrade table gets the matching
`0.15.x -> 0.16.0` row.

tests/fixtures/key_stability/golden_keys.tsv.gz is regenerated so its header
stamps 0.16.0; every row is byte-identical to HEAD's and KEY_FIXTURE_SHA256 is
unchanged, which is what #887's header-excluding digest exists to show.

The RELEASING.md sweep (`grep -rn "0.15.0" --exclude=CHANGELOG.md`) leaves 101
hits: prose about the 0.15.0 cycle, test literals, the meta-benchmark
baseline's measurement record of the 0.15.0 build, and two fixture provenance
stamps. Those are the classes RELEASING.md says stay as written.

Verified on this tree: cargo fmt, clippy on both feature sets, perf_lint, no
pyo3 in the pure dependency tree, cargo test (796 unit + the integration
targets), rustdoc with 0 warnings, pytest 7,636 passed / 42 skipped against a
`maturin develop --release` install that reports 0.16.0, key-stability and
version-consistency 34/34, the four glue pins resolving to disarm 0.16.0 under
the CI patch redirect, and the built wheel installed into a clean venv reporting
0.16.0. Docs and lint on the same install: ruff check and format at the pinned
0.16.5, mypy, the language-consistency audit, toolchain pins, `mkdocs build
--strict`, run_doc_tests 42/42, check_doc_claims, and check_docs_against_release
all pass.

Tier 3 was run on origin/main at 3cc21fb before this branch was cut, and was
green throughout: the exhaustive trio, exhaustive_confusables, the lib-level
ignored set and `pytest -m formal`.

No tag and no GitHub Release are created with this change.

Refs #902

Assisted-by: Claude Fable 5.1

Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants