fix(data): the ICANN Latin second-level LGR homoglyph pairs (#831) - #847
Merged
Conversation
ICANN's *Reference LGR for the Second Level, Latin script* (25 October 2024)
defines 25 variant sets over a 231-element repertoire, expanding to 51 blocked or
fallback pairs. 23 carry the Latin Generation Panel's own comment "Glyphs either
homoglyph or nearly identical". `canonicalize` collided 2 of them.
All 19 that a single code point can express now collide, and the four hostname
rows in the issue are closed:
ważne.pl / waźne.pl gmaġl.com / gmaģl.com
lýs.com / lỳs.com gǝnc.com / gənc.com
This was never a defect in the fold. These are SAME-SCRIPT Latin-to-Latin pairs,
and disarm's confusable data is cross-script — most of the code points are not
TR39 sources at all, so the two sets never met. The rows come from a third data
file with its own admission criterion, beside the cross-script supplement (#342)
and the attested rows (#597).
Both decisions the issue raised are settled, and settling them made this smaller.
THE TARGETS ARE NOT ASCII. Folding `ż` to `z` merges it with the bare letter,
which this LGR does not block — the over-collapse `search_key` already commits at
1,534 non-LGR merges — so the target must be the other member of the pair. The
issue proposed folding to the lower code point pairwise. That is not a function
here: `ỉ` appears in three pairs and would need three targets, `ỷ` in two. And it
regresses one row — it makes `ə` the SOURCE and overwrites its existing TR39 fold
to `e`, undoing a fold that already reaches ASCII, in the very pair the issue
flags as an inconsistency.
Read as equivalence classes instead: 16 classes over 35 code points, with each
representative taken from the class's existing ASCII fold where one exists and the
lowest code point otherwise. Two classes resolve to ASCII, 14 to a non-ASCII Latin
letter, and the `ǝ`/`ə` inconsistency is repaired rather than entrenched.
`build.rs` asserts what makes a non-ASCII target safe rather than trusting it: the
value must be Latin, and it must not itself be a source. All 2,290 rows were ASCII
before and nothing checked it, which is the weakest state to change an invariant
from. Both conditions verified to fail the build — a Greek target and a chaining
target each produce their own message.
BUILD.RS:216 IS NOT RELAXED. The contraction table is the wrong home for a reason
that decides it before the assert is reached: `contraction::contract` is called
only from `src/hostname.rs` behind `if contractions`, so it is unreachable from
`normalize_confusables` and putting `n̄`→`ñ` there would not collide the pair —
this issue's whole claim. The two multi-code-point rows are dropped at a cost of 2
of 23, and the class they belong to is #836: six Latin bases where tilde and
macron disagree on precomposition, which is where a sequence mechanism should be
designed.
`tests/test_lgr_pairs.py` asserts both directions, and the second matters more.
The LGR's other 23 pairs are commented "Required for use with Common LGR" —
transitivity artefacts of running it beside the Greek and Cyrillic rulesets — and
its own Variants section says they can be removed when it is used standalone.
Folding `u`/`ü` or `a`/`á` would strip legitimate diacritics from every language
that uses them, so those are pinned as MUST NOT collide. That is what stops a
later well-meaning import of the whole variant set.
Latin table 2,273 -> 2,290; the five documented counts and the key fixture move
with it, and `docs/provenance.md` gains the LGR as a source with its version date.
Closes #831
Refs #336, #342, #597, #715, #801, #836, #762
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Assisted-by: Claude Code:claude-opus-5
|
📄 Docs preview: https://31c70068.disarm-docs.pages.dev |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are a few concrete correctness/documentation issues in the changed code/docs (notably the misplaced doc comments and overly-broad “Latin letter” block check in build.rs, plus mismatched test documentation) that should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR extends disarm’s confusable-folding data to incorporate ICANN’s Reference LGR for the Second Level, Latin script (25 Oct 2024) “homoglyph or nearly identical” same-script Latin pairs, so canonicalize/normalize_confusables collide the intended LGR homoglyph pairs without importing Common-LGR transitivity pairs.
Changes:
- Add an ICANN LGR-driven override channel (
confusables_lgr.tsv) into the confusables generation pipeline and regenerate the Latin confusables table. - Add Python regression tests that assert the LGR homoglyph pairs collide while the Common-LGR-only transitivity pairs remain distinct.
- Update docs/changelog and introduce build-time assertions for safety around newly permitted non-ASCII Latin targets.
File summaries
| File | Description |
|---|---|
| tests/test_lgr_pairs.py | New pytest coverage for ICANN LGR homoglyph collisions and explicit non-collision of Common-LGR transitivity pairs. |
| src/tables/data/confusables_to_latin.tsv | Regenerated to-Latin confusables table with added LGR-driven rows (and updated totals). |
| src/tables/confusables_data.rs | Update documented mapping counts to reflect regenerated tables. |
| scripts/gen_confusables.py | Add confusables_lgr.tsv loader and merge LGR overrides into the Latin override map. |
| python/disarm/_api.py | Update API docstring counts for Latin confusable mappings. |
| docs/user-guide/confusables.md | Update mapping counts in user guide. |
| docs/provenance.md | Record ICANN LGR as a new provenance source with its version date. |
| docs/limitations.md | Document the LGR inclusion, non-ASCII target rationale, and associated safety assertions. |
| docs/architecture/data-tables.md | Update table entry count for Latin confusables. |
| data/confusables_lgr.tsv | New curated LGR-derived same-script Latin override data file with provenance and admission criteria. |
| CHANGELOG.md | Add upgrade note and changelog entry describing behavioral impact of the new LGR rows. |
| build.rs | Add assertions for non-ASCII Latin targets and introduce a Latin-block predicate helper. |
Review details
Suppressed comments (1)
tests/test_lgr_pairs.py:117
test_the_targets_are_one_stepclaims it asserts the fold “does not chain”, but the body only checks idempotence (fixed-point behavior) by applyingnormalize_confusablestwice. Either strengthen the assertion to actually check “no chaining”, or update the docstring to describe the idempotence property being tested.
def test_the_targets_are_one_step() -> None:
"""No LGR target is itself a source, so the fold does not chain.
`build.rs` asserts this at compile time; asserting it here too states the property in
the place a reader looks for behaviour rather than for build wiring.
"""
- Files reviewed: 12/13 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Three findings on #847, all correct. is_latin_letter had read_char_str_tsv's doc comment fused onto it -- the helper was inserted between a doc block and the function it described. Both are back where they belong. It also tested block membership, not letterhood, and 0x00C0..=0x024F is not all letters: U+00D7 MULTIPLICATION SIGN and U+00F7 DIVISION SIGN sit inside it. It now requires is_alphabetic as well, which immediately caught a real row -- U+2797 HEAVY DIVISION SIGN maps to a non-Latin, non-letter target, and the block range was the only reason the build accepted it. That row is legitimate (both are Sm, and ASCII / is a solidus and means something else), so it is admitted by name through COMMON_SCRIPT_TARGETS rather than as a side effect of a range. Script=Common characters belong to no script, so the "stay inside the script this table folds toward" rule has nothing to say about them; they are not the Greek or Cyrillic target the assert exists to reject. Admitting one is now a decision with a name on it. docs/architecture/data-tables.md now names ICANN LGR (#831) among the sources of the Latin table. The LGR rows are merged by scripts/gen_confusables.py at generation time, so they are inside the published 2,290 and test_doc_table_counts.py was already holding the figure -- the description was the only thing stale. HOMOGLYPH_PAIRS said it was written as escapes and was written as literals. Now escapes, which is what the comment's own reasoning asks for: several of these pairs are indistinguishable from each other on the page, and that is the entire reason the file exists. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq
enabled auto-merge (squash)
September 1, 2026 07:03
Signed-off-by: Richard Quinn <quinn.richard@gmail.com> # Conflicts: # CHANGELOG.md # tests/fixtures/key_stability/golden_keys.tsv.gz
Signed-off-by: Richard Quinn <quinn.richard@gmail.com> # Conflicts: # CHANGELOG.md # tests/fixtures/key_stability/golden_keys.tsv.gz
Signed-off-by: Richard Quinn <quinn.richard@gmail.com> # Conflicts: # python/disarm/_api.py
raeq
added a commit
that referenced
this pull request
Sep 1, 2026
The gate caught its own page: #847's 17 LGR rows and #849's Arabic and Hebrew tables moved the count from 227 of 313 to 226 of 312 between writing the entry and merging main. The detector's share is unchanged at 97.8%. That is the gate working rather than a problem with it -- a figure that moves when the tables move is exactly what test_doc_table_counts.py and test_confusable_residue_docs.py exist to hold, and this one now has the same treatment. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq
added a commit
that referenced
this pull request
Sep 1, 2026
Three findings, all correct, all pre-existing in #847 rather than introduced by the restore -- which is a fair reading of what a restore is for: the code got a second review it would not otherwise have had. load_lgr did not strip the target column while stripping the other two, and while load_supplement and load_attested strip theirs. A trailing space in the TSV would have become part of the fold target. Worth noting the asymmetry is deliberate elsewhere: build.rs's read_char_str_tsv explicitly does NOT trim, because in confusables_to_latin.tsv a trailing space can be the whole value (U+30FB folds to one). An LGR target is always a Latin letter, so stripping is right here and wrong there. The data file's header said "the 19 qualifying pairs" nine lines after saying 23 qualify. Both numbers are true of different things: 23 pairs meet the admission criterion, 19 of those can be expressed by a single code point, and the file holds 17 rows because the 19 collapse into equivalence classes. The header now says which is which. limitations.md hard-coded "the 23 pairs ... are imported", which is a count the page has to keep in step with data it does not own -- and was already imprecise, since 19 are expressible rather than 23. It now describes the criterion, which is the durable statement: the Latin Generation Panel's own comment, a judgement by the people who wrote the registry. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq
added a commit
that referenced
this pull request
Sep 1, 2026
* fix: restore #847, which #851 reverted in full (#831) I broke this. #851 was collapsed to one commit with `git reset --soft origin/main` at a moment when origin/main had moved ahead of the working tree, so the commit recorded the DIFFERENCE -- which included deleting every file #847 had added that the branch had never seen. All fourteen files: 140 lines of build.rs, data/confusables_lgr.tsv, tests/test_lgr_pairs.py, the seventeen ICANN LGR rows in confusables_to_latin.tsv, load_lgr in the generator, and the documented count in four places. The consequence was behavioural, not cosmetic: canonicalize("ż") stopped equalling canonicalize("ź"), so the same-script Latin homoglyph pairs #831 closed had reopened and the four hostname rows that issue names were live again. Nothing caught it, and that is the part worth fixing rather than the files. tests/test_lgr_pairs.py covers this class properly and was deleted by the same commit, so it could not fail. The doc-count gates could not fire either, because the counts were reverted along with the table they check. A 40-row fixture diff was the only trace and it was applied rather than questioned -- by me. Restored file by file rather than by reverting the revert, which was not available: #851 carried the legitimate has_bidi_control work in the same squashed commit. Ten of the fourteen had no later commits other than the revert and were taken verbatim from 79c522f; docs/limitations.md and python/disarm/_api.py had later work and took only #847's own hunks. The key fixture is regenerated, and the 40 rows it moves are the LGR collisions coming back. tests/test_no_silent_revert.py checks the ARTIFACTS rather than the behaviour: a bundled data file must exist and something must reference it, and build.rs must still carry #831's two safety asserts. Behaviour tests travel with the feature and vanish with it; a data file's absence is a build-level fact a deleted test cannot hide. Mutation-checked by moving the file aside. Refs #831, #847, #851 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> * fix: review findings on the restore (#870) Three findings, all correct, all pre-existing in #847 rather than introduced by the restore -- which is a fair reading of what a restore is for: the code got a second review it would not otherwise have had. load_lgr did not strip the target column while stripping the other two, and while load_supplement and load_attested strip theirs. A trailing space in the TSV would have become part of the fold target. Worth noting the asymmetry is deliberate elsewhere: build.rs's read_char_str_tsv explicitly does NOT trim, because in confusables_to_latin.tsv a trailing space can be the whole value (U+30FB folds to one). An LGR target is always a Latin letter, so stripping is right here and wrong there. The data file's header said "the 19 qualifying pairs" nine lines after saying 23 qualify. Both numbers are true of different things: 23 pairs meet the admission criterion, 19 of those can be expressed by a single code point, and the file holds 17 rows because the 19 collapse into equivalence classes. The header now says which is which. limitations.md hard-coded "the 23 pairs ... are imported", which is a count the page has to keep in step with data it does not own -- and was already imprecise, since 19 are expressible rather than 23. It now describes the criterion, which is the durable statement: the Latin Generation Panel's own comment, a judgement by the people who wrote the registry. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> --------- Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
raeq
added a commit
that referenced
this pull request
Sep 1, 2026
I reported three measurements as moving the wrong way between 0.14.1 and 0.15.0
and wrote that nothing in the milestone explained them. That was wrong on both
counts: every one is attributable, and none is a regression. Reporting movement
I had not investigated, and then asserting it was unexplained, is the failure the
harness exists to prevent.
**nonascii_folded could not tell folding from deleting from naming.** The metric
is (before - after) / before over non-ASCII code points, so any way a character
stops being non-ASCII scores the same. disarm 0.14.1 turned `a — b` into
`a em dash b`, and the metric called that coverage:
0.14.1 strip_obfuscation("a — b") -> 'a em dash b'
0.15.0 strip_obfuscation("a — b") -> 'a — b'
So the benchmark was crediting #757 — the same defect that turned `film's` into
`film right apostrophe s` — and then charged #803 for fixing it. Five more cases
were worse than cosmetic: 0.14.1 stripped the negation overlay, turning
`x ≠ y` into `x = y`, and the metric scored #749's fix as a loss too.
`classify_removal` now splits it into folded / deleted / named, measured inside a
stable ASCII carrier because the naming fires on text rather than on a lone code
point. The line between a fold and a name is words, not length: `½` to `1/2` is a
compatibility fold, `—` to `em dash` is a description.
**The gap list was scored by length, not accuracy.**
`visible_to_coverage_introspection` counted pairs present in
`unmapped_confusables()` and scored higher as better, so the only way to reach
100% was to map nothing — covering a pair removed it from the list and lowered
the score. Measured across the release: 4,384 -> 4,330 entries, 54 leaving and
**none joining**, all Cyrillic ICANN LGR rows from #847/#870. Pure coverage gain,
reported as 21.8% -> 19.1%. Replaced by `gaps_not_named` (misses the list does
not carry) and `named_but_covered` (stale entries), neither improvable by
refusing to map anything.
**Also closes #759.** `adversarial_eval.evaluate` hardcoded `strip_obfuscation`,
so the corpus rate and the removal split were measuring different surfaces — the
same asymmetry the coverage and cost axes had. It now takes the declared
sanitizer.
With both fixed, the two youtube-spam movements vanish: 0.14.1 and 0.15.0 are
identical on that corpus at 59.8% removed, 39 folded, 5 deleted, 0 named. Of 39
measurements that move across the release, 2 now go the worse way, and both are
explicable — `gaps_not_named` rises because improved coverage concentrates the
residual gaps, and `identity_retention` moves below 0.1%.
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Assisted-by: Claude:claude-opus-5[1m]
raeq
added a commit
that referenced
this pull request
Sep 2, 2026
I reported three measurements as moving the wrong way between 0.14.1 and 0.15.0
and wrote that nothing in the milestone explained them. That was wrong on both
counts: every one is attributable, and none is a regression. Reporting movement
I had not investigated, and then asserting it was unexplained, is the failure the
harness exists to prevent.
**nonascii_folded could not tell folding from deleting from naming.** The metric
is (before - after) / before over non-ASCII code points, so any way a character
stops being non-ASCII scores the same. disarm 0.14.1 turned `a — b` into
`a em dash b`, and the metric called that coverage:
0.14.1 strip_obfuscation("a — b") -> 'a em dash b'
0.15.0 strip_obfuscation("a — b") -> 'a — b'
So the benchmark was crediting #757 — the same defect that turned `film's` into
`film right apostrophe s` — and then charged #803 for fixing it. Five more cases
were worse than cosmetic: 0.14.1 stripped the negation overlay, turning
`x ≠ y` into `x = y`, and the metric scored #749's fix as a loss too.
`classify_removal` now splits it into folded / deleted / named, measured inside a
stable ASCII carrier because the naming fires on text rather than on a lone code
point. The line between a fold and a name is words, not length: `½` to `1/2` is a
compatibility fold, `—` to `em dash` is a description.
**The gap list was scored by length, not accuracy.**
`visible_to_coverage_introspection` counted pairs present in
`unmapped_confusables()` and scored higher as better, so the only way to reach
100% was to map nothing — covering a pair removed it from the list and lowered
the score. Measured across the release: 4,384 -> 4,330 entries, 54 leaving and
**none joining**, all Cyrillic ICANN LGR rows from #847/#870. Pure coverage gain,
reported as 21.8% -> 19.1%. Replaced by `gaps_not_named` (misses the list does
not carry) and `named_but_covered` (stale entries), neither improvable by
refusing to map anything.
**Also closes #759.** `adversarial_eval.evaluate` hardcoded `strip_obfuscation`,
so the corpus rate and the removal split were measuring different surfaces — the
same asymmetry the coverage and cost axes had. It now takes the declared
sanitizer.
With both fixed, the two youtube-spam movements vanish: 0.14.1 and 0.15.0 are
identical on that corpus at 59.8% removed, 39 folded, 5 deleted, 0 named. Of 39
measurements that move across the release, 2 now go the worse way, and both are
explicable — `gaps_not_named` rises because improved coverage concentrates the
residual gaps, and `identity_retention` moves below 0.1%.
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Assisted-by: Claude:claude-opus-5[1m]
raeq
added a commit
that referenced
this pull request
Sep 3, 2026
I reported three measurements as moving the wrong way between 0.14.1 and 0.15.0
and wrote that nothing in the milestone explained them. That was wrong on both
counts: every one is attributable, and none is a regression. Reporting movement
I had not investigated, and then asserting it was unexplained, is the failure the
harness exists to prevent.
**nonascii_folded could not tell folding from deleting from naming.** The metric
is (before - after) / before over non-ASCII code points, so any way a character
stops being non-ASCII scores the same. disarm 0.14.1 turned `a — b` into
`a em dash b`, and the metric called that coverage:
0.14.1 strip_obfuscation("a — b") -> 'a em dash b'
0.15.0 strip_obfuscation("a — b") -> 'a — b'
So the benchmark was crediting #757 — the same defect that turned `film's` into
`film right apostrophe s` — and then charged #803 for fixing it. Five more cases
were worse than cosmetic: 0.14.1 stripped the negation overlay, turning
`x ≠ y` into `x = y`, and the metric scored #749's fix as a loss too.
`classify_removal` now splits it into folded / deleted / named, measured inside a
stable ASCII carrier because the naming fires on text rather than on a lone code
point. The line between a fold and a name is words, not length: `½` to `1/2` is a
compatibility fold, `—` to `em dash` is a description.
**The gap list was scored by length, not accuracy.**
`visible_to_coverage_introspection` counted pairs present in
`unmapped_confusables()` and scored higher as better, so the only way to reach
100% was to map nothing — covering a pair removed it from the list and lowered
the score. Measured across the release: 4,384 -> 4,330 entries, 54 leaving and
**none joining**, all Cyrillic ICANN LGR rows from #847/#870. Pure coverage gain,
reported as 21.8% -> 19.1%. Replaced by `gaps_not_named` (misses the list does
not carry) and `named_but_covered` (stale entries), neither improvable by
refusing to map anything.
**Also closes #759.** `adversarial_eval.evaluate` hardcoded `strip_obfuscation`,
so the corpus rate and the removal split were measuring different surfaces — the
same asymmetry the coverage and cost axes had. It now takes the declared
sanitizer.
With both fixed, the two youtube-spam movements vanish: 0.14.1 and 0.15.0 are
identical on that corpus at 59.8% removed, 39 folded, 5 deleted, 0 named. Of 39
measurements that move across the release, 2 now go the worse way, and both are
explicable — `gaps_not_named` rises because improved coverage concentrates the
residual gaps, and `identity_retention` moves below 0.1%.
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Assisted-by: Claude:claude-opus-5[1m]
raeq
added a commit
that referenced
this pull request
Sep 3, 2026
I reported three measurements as moving the wrong way between 0.14.1 and 0.15.0
and wrote that nothing in the milestone explained them. That was wrong on both
counts: every one is attributable, and none is a regression. Reporting movement
I had not investigated, and then asserting it was unexplained, is the failure the
harness exists to prevent.
**nonascii_folded could not tell folding from deleting from naming.** The metric
is (before - after) / before over non-ASCII code points, so any way a character
stops being non-ASCII scores the same. disarm 0.14.1 turned `a — b` into
`a em dash b`, and the metric called that coverage:
0.14.1 strip_obfuscation("a — b") -> 'a em dash b'
0.15.0 strip_obfuscation("a — b") -> 'a — b'
So the benchmark was crediting #757 — the same defect that turned `film's` into
`film right apostrophe s` — and then charged #803 for fixing it. Five more cases
were worse than cosmetic: 0.14.1 stripped the negation overlay, turning
`x ≠ y` into `x = y`, and the metric scored #749's fix as a loss too.
`classify_removal` now splits it into folded / deleted / named, measured inside a
stable ASCII carrier because the naming fires on text rather than on a lone code
point. The line between a fold and a name is words, not length: `½` to `1/2` is a
compatibility fold, `—` to `em dash` is a description.
**The gap list was scored by length, not accuracy.**
`visible_to_coverage_introspection` counted pairs present in
`unmapped_confusables()` and scored higher as better, so the only way to reach
100% was to map nothing — covering a pair removed it from the list and lowered
the score. Measured across the release: 4,384 -> 4,330 entries, 54 leaving and
**none joining**, all Cyrillic ICANN LGR rows from #847/#870. Pure coverage gain,
reported as 21.8% -> 19.1%. Replaced by `gaps_not_named` (misses the list does
not carry) and `named_but_covered` (stale entries), neither improvable by
refusing to map anything.
**Also closes #759.** `adversarial_eval.evaluate` hardcoded `strip_obfuscation`,
so the corpus rate and the removal split were measuring different surfaces — the
same asymmetry the coverage and cost axes had. It now takes the declared
sanitizer.
With both fixed, the two youtube-spam movements vanish: 0.14.1 and 0.15.0 are
identical on that corpus at 59.8% removed, 39 folded, 5 deleted, 0 named. Of 39
measurements that move across the release, 2 now go the worse way, and both are
explicable — `gaps_not_named` rises because improved coverage concentrates the
residual gaps, and `identity_retention` moves below 0.1%.
Signed-off-by: Richard Quinn <quinn.richard@gmail.com>
Assisted-by: Claude:claude-opus-5[1m]
raeq
added a commit
that referenced
this pull request
Sep 3, 2026
… used (#875) * feat: a meta-benchmark over the externally produced benchmarks 0.15.0 used The 0.15.0 cycle found and verified defects against roughly thirty artifacts produced outside this repository — corpora released with papers, public spam and phishing datasets, normative Unicode/IETF/ICANN tables, published CVEs, released tokenizers and a third-party labelled benchmark. Each measurement lived in its own script, recorded as a gist on the issue it produced, and nothing re-ran any of them. #732 states the consequence directly: one public corpus produced five findings and nothing in the repo will notice when any of them moves. `benchmarks/meta` consolidates the selection, the provenance record, the scoring protocol and the report into one harness that runs n of m: python -m benchmarks.meta --list python -m benchmarks.meta --run --only-available python -m benchmarks.meta --run --select 'uts39-*' --report out.md 35 benchmarks are registered across six tiers: normative tables, published CVEs, academic corpora, public datasets, released model artifacts, and third-party labelled benchmarks. The harness supplies the runner and supplies no vectors — a library that writes its own benchmark grades its own homework. The bias boundary is machine-checkable rather than promised. `Provenance.external` is a field, the runner tags every outcome with it, the report renders external and introspective results in separate sections, and a test asserts that no external suite is scored against a file disarm generated. That last one is not hypothetical: `data/confusables_lgr.tsv` is an ideal-looking local oracle for the ICANN suite, and it is also the file the shipped fold was built from, so scoring the fold against it returns 100% by construction. The suite requires ICANN's published LGR or reports nothing. A seventh tier, `introspective`, holds the sweeps whose right answer only disarm can supply (#719, #723/#751, #805/#806/#807, #834). They are registered so the 0.15.0 record has no hole, marked `external=False`, excluded unless `--include-introspective` is passed, and never folded into an external total. Three things are reported per suite and kept apart: what the benchmark found during the cycle (historical, quoted from the issue, never edited to match a fresh run), how it is measured (methodology), and what it measures now. The gap between the first and the third is where a landed fix shows up as a number, and several already do — UTS #39 §5.3 was unimplemented under #777 and now reports 75 of 75 numbering systems; `has_bidi_conflict` was neutral to 1,786 of 3,018 strong-RTL code points under #773 and now reaches all 3,018. A suite whose artifact is absent reports SKIPPED with the variable to set. An absent corpus is not a passing corpus, and most academic corpora are deliberately not vendored: copying an attack corpus into this repository would make it disarm's corpus, and a corpus disarm owns is a corpus disarm can be tuned against. The #39/#40 guardrail carries over unchanged — these are measuring instruments, never optimization targets. Drift is reported and never gated. Baselines are keyed by suite *and* population, because a ratio over 4,000 code points and one over 150,000 are not comparable; the report marks such a row rather than subtracting it. `--limit` samples by stride rather than truncating — the first N code points of any sorted domain are Latin, Greek and Cyrillic, the best-covered part of every table here, so a truncating quick pass would disagree with a full one for the wrong reason. Also fixed while here, both found by the harness against its own results: - `ucd-scripts` counted unassigned code points in its denominators, so agree + contradict + silent did not sum to the population. - `benchmarks/adversarial_eval/corpora.py` typed `_guess_text_column` as taking `list[str]` while both call sites pass `csv.DictReader.fieldnames`, a `Sequence[str]`. Refs #732, #736, #759 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: score other tools on the meta-benchmark, and pair every score with its cost Four changes, each closing a way the harness could report a number that did not mean what it appeared to. **The suites did not reproduce the measurements they cited.** The report's headline is a finding/now pair, and nothing checked that the two halves answered the same question. `ascii-producing-steps` cited #719's 261-via-NFKC and 232-via-fold and measured 266 and 253, because it used a looser punctuation set, a bare detection probe and an assigned-only domain. A reader would have read a 21-code-point change in behaviour off a change in method. Each suite now recomputes its published script's exact quantity alongside its own sweep, pinned to what that script printed at v0.14.1. Eighteen pins across six suites, all verified against a real v0.14.1 build in its own worktree; #719 now reproduces 493/261/232/174/76 exactly, and #713 lands on 684. Pins are taken from *executing* each gist, not from its docstring — the segmentation census header says `total=36` and running it prints `37`. `Reproduction.matches` is the only thing that licenses reading a finding and a measurement as a before/after, and the report says so in both directions. **Only disarm was ever scored.** `--subject` runs the suites against the pinned comparator environment in requirements/bench.txt (ftfy, unidecode, text-unidecode, anyascii, decancer) plus CPython's normalization as the floor, so an absolute figure becomes a column. A suite measuring a disarm-specific surface skips other subjects rather than scoring them zero. The first cross-tool result was itself a bad metric: scoring "the tool changed the code point" put unidecode at 99.9%, because a transliterator rewrites everything. Coverage is now "source and its UTS #39 target land on one form". **A tool that deleted everything was the best tool in the registry.** Both sides of every comparison became identical, so it folded 100% of UTS #39 onto target and closed every equivalence class. Collisions now require a non-empty shared form, and XMR requires something to survive. `null-baseline` and `identity` are registered as permanent controls: a control that is meant to fail is the only thing that proves a metric can fail, and a test asserts both score zero. **Coverage had no paired cost.** `corruption-cost` measures the other direction — code points destroyed, characters retained, injectivity, and alteration of pure ASCII that has nothing to fix. Labelled corpora get the same axis free from their own `clean` column, which is by the corpus author's definition text needing no repair. Both ends are reported with the surface names at each end, and key builders are separated: `sort_key` and `catalog_key` are many-to-one by contract, and counting that as damage would rank a tool with no key builder as safer. Every run now records its method — subject, domain, predicates invoked, parameters, the sha256 of the artifact read, and the Unicode/UCD versions. The baseline is keyed by subject as well as suite and carries the table versions, so a UCD bump no longer reads as a regression in the tool. Review findings from #875, each with a test that fails without the fix: - `report.fmt()` treated `of=0` as "no denominator", hiding exactly the empty-population case the report exists to show. - Two tests depended on some registry suite happening to be unavailable, so a populated `$DISARM_META_CACHE` would fail them. They now use an explicit always-missing double. - `manufactured_from_fullwidth` read a cumulative counter inside the row loop, so once one row fired every later row counted too. It was also never reported. Also fixed: `ucd-scripts` counted unassigned code points in denominators that excluded them, so agree + contradict + silent did not sum to the population. Refs #732, #736, #759 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * chore: re-record the baseline against the rebased tree #873 changed which fold targets are themselves sources, which moves five measurements across the confusable suites. All five are inside the noise floor, so the drift table reported them and flagged none — but the baseline should come from the tree it ships with rather than from the one before the rebase. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: provision corpora automatically, and rank subjects only when the battery can carry a rank **Provisioning.** Twenty-one suites were unrunnable because every academic corpus was declared a manual download. The argument for that — copying an attack corpus into this repository would make it disarm's corpus — is about *vendoring* and does not reach *caching*. A file pulled from its upstream URL into a scratch directory is still the upstream's corpus. The conflation cost twenty-one suites for no benefit. A run now provisions what the selection needs before any suite executes, and leaves untouched anything already on disk, so an operator who placed a specific revision keeps it. `--offline` never reaches the network; `--refresh` forces a re-fetch. Every download is recorded in a manifest with URL, sha256, size and licence, so a figure can be traced to the bytes it came from. Nothing fetched is ever committed. Six upstreams were verified to exist before being wired, rather than assumed: - confusable-bench.v1 (namespace-guard, MIT) — 140 rows, 120/20, matching #736 - confusable-vision weights v2 (CC-BY-4.0) — 4,174 pairs, matching its own meta - Trojan Source PoC files (MIT) — 311 real source files, not one-line fragments - untrace testdata (MIT) — 64 techniques under the rival's own taxonomy - UCD DerivedCoreProperties.txt — 405 Default_Ignorable, matching #770 - ICANN Latin second-level LGR Suites with no *verified* upstream stay manual and say so. Inventing a URL would imply a download nobody has shown to exist. **An empty parse is now an error.** `icann-lgr-latin` reported `blocked_pairs: 0` while its two-table join was wrong, and 0/0 reads as perfect agreement rather than as a fault. A present artifact yielding nothing is a parser fault. That join is still unfinished, so the suite errors honestly instead of scoring. **Leaderboard.** Composite of discrimination-weighted z-scores, every step a named method: corrected item-total correlation for the weights (classical test theory — Crocker & Algina ch. 14), item parcelling within a suite (Little et al. 2002), Bradley-Terry by Hunter's MM algorithm (2004) for a rank that uses only pairwise order, Cronbach's alpha (1951) and Kendall's W for whether the battery is coherent at all, and bootstrap intervals over the benchmark set. IRT is the method of record for discrimination weighting (Rodriguez et al., ACL 2021) and is deliberately not fitted: single-digit respondents cannot support 2PL estimates, and fitting one would look more rigorous while being less so. Two distortions found and fixed while building it. `corruption-cost` supplied seven near-duplicate items at r≈0.945 and took seven times the weight — hence parcelling. And letting `null-baseline` into the standardisation compressed every real tool toward the mean, so the scale is fitted on the tools and the controls are placed on it. **The leaderboard refuses to publish.** On the current battery: 3 directed benchmarks against a floor of 5, Cronbach's alpha 0.05 against the conventional 0.70 (Nunnally 1978), and no two subjects with non-overlapping intervals. All three interlocks fire, no ranking is published, and the composites are printed only so the shortfall is auditable. A leaderboard that cannot fail is not a measurement. Refs #732, #736, #759 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: a subject is name@version, and two builds of one tool can compete Subject identity was the bare tool name, so two builds of one tool would have collided everywhere it mattered: overwriting each other in the baseline, sharing a column in the comparison table, and being averaged together in the leaderboard. `disarm@0.14.1` against `disarm@0.15.0` is the comparison most worth making here, and it was the one the harness could not express. Identity is now `name@version` throughout — baseline keys, leaderboard subjects, comparison columns, drift rows, the method record and every rendered heading. A test asserts no report can name a tool without its version. A compiled extension cannot be imported twice in one process, so two builds cannot both be live. `--merge` folds earlier JSON runs into the comparison and the leaderboard, which is how each version is measured in its own worktree and then ranked against the other. Also in this change: the ICANN LGR suite now parses. Its pairs live in per-set Variant Set tables that the repertoire table only references by name, so it is a join and not a scan, and the column layout differs between sets. It recovers 21 of the 23 pairs #831 counts. The shortfall is reported as its own measurement rather than tuned away — a denominator that quietly disagrees with the issue it cites is the exact failure this harness exists to catch, and the two missing rows are most likely continuation rows where a mapping spans two `<tr>`s. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: three more subjects, including the standard's own implementation `disarm` was the only subject with a detect capability, so every detector suite was locked to it and no detection question had a second column. A benchmark with one participant is a description, not a comparison. - `confusable-homoglyphs` (MIT) — detects confusable and mixed-script identifiers from UTS #39 data. The second detector in the registry. - `pyunormalize` (MIT) — NFC/NFD/NFKC/NFKD against its own bundled UCD 17.0.0, while the interpreter's `unicodedata` is 16.0.0. It isolates *table version* from *algorithm*, the one variable the stdlib column cannot vary. - `icu` (PyICU) — `SpoofChecker` implements UTS #39 directly and `Transliterator` covers romanization, which makes it the most informative column available: not another tool but the standard's own implementation. It needs the ICU C++ headers and is not installed here, so it registers as unavailable with the install hint. A missing reference implementation should be visible as missing rather than absent from the list. `uax29-word-joiners` becomes multi-subject on the back of that, taking the battery from three benchmarks to four. It asks two separable questions, so capability handling gained "at least one of" semantics: a subject answers the half it has and the other half is *omitted*, never reported as zero. `confusable-homoglyphs` detects 100% of fragmented words where disarm detects 59.5%, and shows no recovery column at all rather than a misleading 0%. The leaderboard still refuses: four benchmarks against a floor of five, alpha 0.06. Adding subjects cannot fix that — two of the three blockers are about the number and coherence of *benchmarks*, not the number of tools. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: rank each benchmark on its own, and keep partial coverage out of the composite Two ranking problems, both visible the moment more subjects existed. **A subject measured on one benchmark ranked above subjects measured on four.** `confusable-homoglyphs` detects and does not transform, so it participates in a single suite — and came first, because answering fewer questions is not the same as answering them better. Subjects covering less than 75% of the battery are now listed with their coverage and kept out of the ordering entirely. **Every benchmark now carries its own ranking.** The composite needs the benchmarks to measure one construct before averaging them, which is exactly the assumption Cronbach's alpha says this battery fails. Ranking within a single benchmark carries no such assumption, so those tables stand whether or not the composite does — and while the composite is blocked, they are the result. Equal scores share a rank, and a subject absent from a table was not asked that question rather than having scored zero. The composite still refuses: four benchmarks against a floor of five, alpha 0.06, no separated pairs. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: three ways the comparison could be read backwards **`retention` could exceed 100%.** It was `chars_out / chars_in`, so a transliterator mapping one code point to several ASCII characters — `¼` to `1/4`, a CJK ideograph to a syllable — scored 102.6% "retention", which is not a thing. Split into three honest numbers: `length_ratio` (out over in, may exceed 1 and named so it cannot be read as retention), a true `retention` (multiset intersection, bounded at 1, so characters a tool *adds* cannot inflate it), and `max_expansion`. Expansion earns its own measurement because it is a finding here, not a curiosity: #768 measured 18x amplification with no ceiling and #747 found presets manufacturing delimiters the input never contained. `anyascii` reads 102.5% length against 90.7% actual retention, peaking at 3x. **A parcel averaged whatever measurements a subject happened to have.** `unidecode` outranked `disarm` on the word-joiner benchmark while recovering 24.3% to disarm's 43.2% — disarm's average also carried a detection score, and `unidecode` has no detector to be scored on. Subjects answering only part of a benchmark are now listed after its ordering rather than inside it, the same rule already applied one level up for partial battery coverage. **A lower-is-better row read backwards.** The comparison table printed bare percentages, so `unreached` 34.1% next to 44.5% looked like a loss when it is the best score on the row. Directed rows now carry ↑ or ↓, the winning cell is bold, and a census row carries no arrow because it has no better end. Also: `disarm`'s subject identity now carries the build commit — `0.14.1+g5ff5582` — with `.dirty` for an uncommitted tree. `__version__` only moves at release, so every build between two releases reports the older number, and a row reading `disarm@0.14.1` while the extension carries post-0.14.1 code is exactly the mislabelling the versioned-identity rule exists to prevent. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: a control can neither win a row nor hold a rank `identity` was marked as the best value on `altered_but_not_onto_target` with 0.0%, because a tool that never alters anything trivially wins a row scored on altering wrongly. `null-baseline` would take any row scored on leaving things unfolded, by leaving nothing at all. Presenting either as the winner puts the degenerate answer forward as the target — the same failure the non-empty collision rule fixed, resurfacing one layer up in the report. Controls are now excluded from the best-cell calculation, from the composite ordering and from every per-benchmark ranking. Their values stay visible, because a reference line is the point of having them: `identity` sitting above `disarm` on corruption cost is exactly the comparison that axis exists to make. They just cannot occupy a position that asserts they beat something. Two related corrections in the same pass. Partially-measured subjects were already kept out of the ordering but still printed a numeric rank; they now print an em-dash like controls. And a benchmark with fewer than two fully-answered subjects no longer prints an ordering at all — `uax29-word-joiners` is answered in full only by `disarm`, and "1st of 1" dresses up a benchmark nobody else could be asked. With controls out of the ordering, the composite reads disarm first of eight ranked tools. The battery still refuses to publish it: four benchmarks against a floor of five, alpha 0.06, no separated pairs. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: a Pareto leaderboard, and Friedman's test on whether the benchmarks agree The composite was gated on Cronbach's alpha, which asks whether the benchmarks measure one construct. This battery is not built to: coverage and cost are deliberately opposed axes, so a tool that folds more will alter more, and alpha *should* be low. A unidimensional psychometric gate was being applied to a multidimensional measurement problem. Changing the test does not rescue it. Friedman's test — the right diagnostic for a rank aggregation, and one that assumes no common construct — refuses too: chi-square 4.75 against a 0.05 critical value of 14.07 at 7 degrees of freedom, Kendall's W 0.226. The benchmarks genuinely disagree. `ftfy` places 1st, 1st, 7th; `anyascii` places 8th, 6th, 2nd. Tools that preserve text win on cost and lose on class closure; tools that fold aggressively do the reverse. That disagreement is the finding, not a defect, and it has a standard answer. **Pareto dominance** ranks without weighting and without assuming one construct: a tool is on the frontier when nothing beats it on every axis at once. On the current battery four tools are non-dominated — `decancer`, `disarm`, `ftfy`, `stdlib` — and four are strictly beaten: `anyascii` and `unidecode` by `disarm`, `pyunormalize` by `ftfy`, `text-unidecode` by both. It is a partial order rather than a league table, which is the honest shape of the result. The report now also prints how many benchmarks the observed agreement would need to reach significance: **9**, against the 3 available. Friedman's chi-square is k(n-1)W, so this is linear in the benchmark count — and more *tools* make it harder, not easier, by raising the degrees of freedom. Ten attack-corpus suites are already multi-subject and waiting only on corpus data, so the route to a significant overall ranking is provisioning them rather than adding comparators. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: the surfaces that earned coverage were exempt from cost Two accounting biases on the two axes disarm scores highest on, both in its own favour, on its own benchmark. **The surfaces that earned the coverage were exempt from the cost.** `search_key`, `catalog_key` and `sort_key` are inside `PRESETS`, so `transforms()` returned them and the coverage axis scored with them — while the cost axis removed exactly those three via `split_by_intent`. A library that ships key builders collected their coverage for free. The comment there reasoned about the opposite bias, which is also real, and the fix applied produced the inverse. Measured: 1.2 points of the confusable headline was coverage only a key builder earned. **Coverage was a union over every surface a subject happens to expose.** The existential asked "did any of your N entry points get this pair", which rewards shipping many rather than shipping good. disarm exposes 19 transforms against one to five for every other tool, and gained 4.9 points from the union that no other subject could earn — none of them has enough surfaces for a union to differ from its best one. Both axes now score the best *single* non-key surface, which also makes coverage symmetric with cost — the cost side was already per-surface, and that asymmetry is what let the two sets diverge. Key builders are scored in their own role, in their own measurement, where merging is the contract rather than a cost. The winning surface is named and the number of surfaces each subject was allowed is reported, so a reader can see that disarm's score came from one entry point out of thirteen while ftfy's came from one out of two. disarm's confusable coverage moves 65.9% to 61.2% and its class closure 60.7% to 56.6%. It stays on the Pareto frontier; `ftfy` and `pyunormalize` now edge it on confusables. Two more found on the way: - `library_catalog_key_eu`, `search_index` and `scholarly_cyrillic_iso9` are key builders that live among the profiles, so excluding only the three top-level key functions left them scored as text surfaces. `library_catalog_key_eu` was the single most destructive "text" surface in the corruption census, which is precisely what a catalog key should look like. - Raw retention charged a sanitizer for removing private-use, format and control code points, which is the one thing it exists to do. 93.5% of disarm's measured "damage" was Private Use Area removal. The scored measure is now identity retention — letters and symbols only — with raw retention kept as a census. - A surface count of 1 rendered as "100.0%" in the comparison table: the cell formatter could not tell a proportion from a small integer. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: score a configuration, not a library's best surface Turning the union into a max was the same effect, quieter. Measured over 938 UTS #39 pairs, disarm's thirteen non-key surfaces run: profile:llm_guardrail 58.0% canonicalize 53.4% strip_obfuscation 57.1% profile:rag_ingest 45.7% canonicalize_strict 53.5% ml_normalize 39.1% normalize_user_input 53.5% strip_format 0.0% security_clean 53.4% profile:code_context 0.0% Removing the union took off 4.7 points. A further 4.6 remained: the gap between "your best of thirteen" and the one a reader would actually reach for. Every other tool draws from one or two. Three things the spread showed. The winner was `llm_guardrail` — a ten-step application pipeline nobody reaches for to clean a username, scoring a general confusable-coverage axis. Thirteen surfaces are not thirteen capabilities: five score identically because they share one fold and three do not fold confusables at all, so there are about four distinct behaviours. And the asymmetry survived one level down — coverage was a max over surfaces while cost averaged a *worst* and a *gentlest*, so the surface earning the coverage never paid its own cost. Coverage came from `llm_guardrail` while cost averaged `rag_ingest` and `code_context`. The published point described a configuration nobody could deploy. Each subject now declares one surface per role before the run, and coverage and cost are both measured on it: role disarm ftfy unidecode stdlib sanitizer canonicalize fix_text unidecode NFKC key search_key — — — detector is_confusable — — — `canonicalize` because it is the documented general-purpose comparison form — the entry point a reader arrives at, not the one that wins. What best-of-N would have added is now a reported census per subject rather than a disclosed surface count, so the selection effect is a measured line like the other three biases: +3.7 points for disarm, +28.5 for ftfy, whose coverage was coming from `fix_text_NFKC` rather than its documented `fix_text`. Best-of-N does answer a real question — the most a library can do for you — but it is not the question this page asks, and it assumes a reader who already knows which of thirteen surfaces to pick, which is the problem the library exists to solve. The worst and gentlest surfaces are still reported as censuses, because the range a library offers is real information; they are simply not the score. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: record the confusable fold's configuration instead of inheriting it disarm's confusable resolution takes two parameters and I chose neither. Both change the result, both were left at their defaults, and neither was recorded. **target_script defaults to "latin".** Of the 6,565 single-source UTS #39 pairs, only 1,968 (30.0%) have a Latin target: 21.2% target CJK, 14.2% Arabic, 6.1% Hangul. So 70% of the denominator asks a Latin-targeting fold to produce a target it does not aim at, and whatever coverage it gets there comes from the NFKC step rather than the fold. `folded_latin_target` is now reported beside the whole table, and it changes the reading: on the subset a Latin fold is actually aimed at, `decancer` leads at 79.9% and disarm is second at 69.4%, where on the full table disarm leads at 57.5%. disarm accepts latin, cyrillic, arabic and hebrew, and rejects greek — though 159 pairs in the table target Greek. **digit_policy defaults to "numeric"** and differs from "tr39" on 45 code points, in opposite directions: U+0660 ARABIC-INDIC DIGIT ZERO folds to `0` under numeric and to `.` under tr39. Scoring against the TR39 table with disarm's own policy costs it 0.7 points (27.1% against 27.8%), so this inherited default understates it rather than flattering it. The finding underneath both: **neither knob is reachable from the surface being scored.** `canonicalize()` takes no arguments, so the configurable fold lives on `normalize_confusables()`, which is not the entry point a reader arrives at. The method record now carries the whole configuration — target script, digit policy, whether the scored surface exposes them, and where the alternatives live. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: score each confusable target script against the pairs it aims at Every other measurement of the fold scores one target script against the whole UTS #39 table, where 70% of the pairs resolve somewhere it does not aim — so what it measures there is the NFKC step, not the fold. This asks the fair question instead: of the pairs that resolve TO Arabic, how many does the Arabic target reach? target pairs in table resolved by its own profile latin 1,968 (30.0%) 1,373 = 69.8% cyrillic 36 ( 0.5%) 22 = 61.1% arabic 935 (14.2%) 136 = 14.5% hebrew 24 ( 0.4%) 4 = 16.7% greek 159 ( 2.4%) REJECTED #792 added the Arabic and Hebrew targets because intra-RTL confusables had no representation in either shipped table. Measured on their own terms they reach 14.5% and 16.7% — which corroborates #791 (whole equivalence classes dropped when no member is in the target script, 948 of 1,007 strong-RTL sources among them) and #848 (a class whose members are all in the target script is discarded by construction, the keheh/kaf case). Greek is rejected while carrying 159 pairs — more than Cyrillic and Hebrew combined, both of which are supported. The rejection message is also stale: it reads "target_script must be 'latin' or 'cyrillic'" and does not name the two targets #792 added. Pairs are partitioned by the UCD name of the target's first character, which is external. Partitioning with `detect_scripts` would use disarm's own table to decide what disarm's own table should cover. The suite is disarm-locked and says so: no other tool in the registry has a target-script parameter, so this scores four configurations of one library rather than comparing several. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * chore: rebase on 0.15.0, and make the harness survive builds older than itself The versioned identity finally means something: `disarm@0.15.0+g711b802` rather than a build labelled 0.14.1 while carrying post-0.14.1 code. Running against the 0.14.1 reference build errored on all 19 suites, because `detectors()` referenced `has_bidi_control` directly and that arrived during the 0.15.0 cycle. A harness that can only run against the version it was written for cannot make the cross-version comparison the whole versioned-identity design exists for. Surfaces are now resolved by name — presets, profiles, key builders and detectors alike — so one the build lacks is absent rather than fatal, and the Trojan Source suite falls back to the twelve UAX #9 controls when the predicate is missing. With both builds running, 41 measurements moved between 0.14.1 and 0.15.0. The sharpest, all matching the issues that drove them: uts39-mixed-numbers ascii_mixed_flagged 0.0% -> 100.0% (#777) ucd-bidi-class conflict_detected 40.5% -> 100.0% (#773) uts39-augmented disagrees_with_uts39 50.0% -> 0.0% (#776) confusable-bench-v1 recall_has_anomalies 35.8% -> 75.8% (#736) uts39-target-scripts target_scripts_supported 40.0% -> 80.0% (#792) bad-characters clean_corrupted_gentlest 41.2% -> 0.4% (#746) cldr-emoji glossed_by_ml_normalize 97.2% -> 79.9% (#757) `confusable-bench-v1` is the strongest of these: 35.8% is exactly the 0.358 #736 published for `has_anomalies` on that corpus, measured independently here against an externally labelled benchmark. The harness reproduces the issue's own number and then shows it doubling. Three went the other way and are not explained by anything in the milestone: youtube-spam nonascii_folded 81.9% -> 78.8% youtube-spam misses_principled 6 -> 8 confusable-vision visible_to_coverage 21.8% -> 19.1% Fewer non-ASCII code points folded on the UCI corpus, two more distinct addressable misses, and fewer measured pairs visible to `unmapped_confusables()`. Filed as observations rather than diagnoses — the harness reports movement and does not explain it. Baseline re-recorded at 0.15.0 / Unicode 17.0.0, 65 entries. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: two metrics that scored a bug as coverage and a fix as a loss I reported three measurements as moving the wrong way between 0.14.1 and 0.15.0 and wrote that nothing in the milestone explained them. That was wrong on both counts: every one is attributable, and none is a regression. Reporting movement I had not investigated, and then asserting it was unexplained, is the failure the harness exists to prevent. **nonascii_folded could not tell folding from deleting from naming.** The metric is (before - after) / before over non-ASCII code points, so any way a character stops being non-ASCII scores the same. disarm 0.14.1 turned `a — b` into `a em dash b`, and the metric called that coverage: 0.14.1 strip_obfuscation("a — b") -> 'a em dash b' 0.15.0 strip_obfuscation("a — b") -> 'a — b' So the benchmark was crediting #757 — the same defect that turned `film's` into `film right apostrophe s` — and then charged #803 for fixing it. Five more cases were worse than cosmetic: 0.14.1 stripped the negation overlay, turning `x ≠ y` into `x = y`, and the metric scored #749's fix as a loss too. `classify_removal` now splits it into folded / deleted / named, measured inside a stable ASCII carrier because the naming fires on text rather than on a lone code point. The line between a fold and a name is words, not length: `½` to `1/2` is a compatibility fold, `—` to `em dash` is a description. **The gap list was scored by length, not accuracy.** `visible_to_coverage_introspection` counted pairs present in `unmapped_confusables()` and scored higher as better, so the only way to reach 100% was to map nothing — covering a pair removed it from the list and lowered the score. Measured across the release: 4,384 -> 4,330 entries, 54 leaving and **none joining**, all Cyrillic ICANN LGR rows from #847/#870. Pure coverage gain, reported as 21.8% -> 19.1%. Replaced by `gaps_not_named` (misses the list does not carry) and `named_but_covered` (stale entries), neither improvable by refusing to map anything. **Also closes #759.** `adversarial_eval.evaluate` hardcoded `strip_obfuscation`, so the corpus rate and the removal split were measuring different surfaces — the same asymmetry the coverage and cost axes had. It now takes the declared sanitizer. With both fixed, the two youtube-spam movements vanish: 0.14.1 and 0.15.0 are identical on that corpus at 59.8% removed, 39 folded, 5 deleted, 0 named. Of 39 measurements that move across the release, 2 now go the worse way, and both are explicable — `gaps_not_named` rises because improved coverage concentrates the residual gaps, and `identity_retention` moves below 0.1%. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: mine issue #909 for benchmarks — one wireable of eighteen it is whether a downloadable artifact exists and carries a licence. Probing all 18 abstract pages and searching for the on-topic ones by title: wireable 1 exists, no licence 1 no locatable artifact 16 **weaponizing-unicode (arXiv:2010.04382, MIT).** `new_predicted_homoglyphs.txt`, 8,452 code points a triplet-loss model identified from weakly labelled font renderings. Worth having because it is the third independent way of deciding what a homoglyph is: UTS #39 is a curated committee table, confusable-vision is measured by rendering and comparing glyphs, this is learned. Where the three disagree is where the coverage question is actually open — and they disagree a lot. disarm flags 3.1% of the set and confusable-homoglyphs 4.9%. Two things it is scored carefully for. It is a code-point SET, not source->target pairs, so it measures detection rather than folding. And 12.4% of it is Private Use — an artefact of deciding confusability by rendering glyphs — which is excluded from the scored denominator, because a tool that strips PUA handles those for a reason unrelated to confusability. Crediting it would repeat the mistake raw retention made with format characters. The transform column is a census rather than a score: a model's weak label is not authority that a character should be rewritten, and the paper says the labels are weak. **reverse-captcha (arXiv:2603.00164)** is registered and deliberately NOT provisioned. It is the most on-topic corpus in the list — invisible Unicode carrying instructions into an LLM is the channel #742 and #748 describe, and the repository holds a prompt set and graded results — but it carries no licence file, and absence of a licence is not permission. It is listed so the registry records that the corpus was found and why it is unused, rather than leaving a reader to assume it was missed. If the authors add a licence, wire SOURCES and it runs. The other sixteen have no artifact anyone has published. Several are squarely on topic — TAG-block concealment in MCP (2607.05744), RAG-Pull (2510.11195), Unicode watermarking detectability (2512.13325), authorship attribution (2508.15840) — and a GitHub search by title finds nothing for any of them. 2405.14490 was already registered. Registry is now 39 suites. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * fix: reverse-captcha is MIT — a licence in prose is still a licence I recorded this corpus as unlicensed and excluded it. It is MIT, declared under a `## License` heading in the README. The check read GitHub's detected `license` field and looked for a file named LICENSE; neither reads prose, and the result was a false negative that dropped the most on-topic corpus in #909. The corpus is now wired and it is a good one. 50 cases: 40 zero-width injections across four schemes (unhinted, hint-aware, hint-codepoints, hint-full) and **10 benign controls**, so false positives are defined by the corpus author rather than by disarm — the second labelled benchmark in the registry after confusable-bench.v1. Each case carries the answer a clean model gives and the answer a compromised one gives, so the attack has ground truth. It is scored on both halves, because removing the payload while mangling the prompt is not a win: subject detected false-pos payload removed visible intact disarm (canonicalize) 100.0% 0.0% 100.0% 0.0% anyascii — — 100.0% 100.0% confusable-homoglyphs 100.0% 100.0% — — stdlib / ftfy — — 0.0% 100.0% Two readings worth stating. `confusable-homoglyphs` detects every attack and also fires on every control, which is not detection. And disarm removes every payload while preserving none of the visible prompts, because the declared sanitizer is `canonicalize` and it collapses whitespace — the #745/#746 finding arriving from a new direction, on somebody else's corpus. `anyascii` is the only subject that does both. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: three benchmarks derived from arXiv source bundles My mining of #909 checked abstract pages and searched GitHub. It never fetched `arxiv.org/src/`, where a paper's LaTeX bundle often carries the construction even when no dataset was released. Three of the sixteen I had written off as having "no locatable artifact" are buildable from their own source. None of these redistributes data. Each derives its vectors from a construction the paper publishes, which is the same footing as the fullwidth chat-template spellings: quoted specification, generated vectors. **mcp-tag-block-concealment (arXiv:2607.05744).** Listing 1 gives the encoder the paper calls "reproduced verbatim": `chr(0xE0000 + (ord(c) & 0x7F))` per ASCII byte, prefixed with the visible label "Formats code neatly.". Of the eight techniques it tests, T7 is the only one evading both the base and the revised approval view. disarm detects the concealed vector and not the plain-ASCII one (correctly — a plain instruction is not a Unicode problem), removes the payload and keeps the label. `anyascii` DECODES the hidden instruction into readable text, which is reported as a census because it is neither clearly right nor wrong. **rag-pull-invisibles (arXiv:2510.11195).** §6 specifies the carrier set by category — 382 characters, 262 Mn and 120 Cf — and the paper publishes its own defence table: stripping that set takes top-1 attack success from 50.2% to 0.0%, category stripping does the same, and **NFKC leaves it at 50.2%**. Measured here, `stdlib` and `pyunormalize` remove 1.5% of the category domain, so the published claim holds and the suite is measuring what the paper measured. A low score is not automatically bad: Mn is where legitimate diacritics live, and the tools scoring 100% are transliterators that flatten everything to ASCII. **zero-width-stylometry (arXiv:2508.15840).** The bundle ships the authors' Python: U+200B encodes 0, U+200C encodes 1, U+200D separates letters, U+FEFF terminates. A fourth invisible channel with a different purpose from the others here — stylometric evasion rather than injection — and THREAT_MODEL names neither. arXiv:2512.13325 (Unicode watermarking) has a source bundle but no concrete construction in it: three mentions of "zero-width" and no scheme. A citation, not a benchmark. Registry is 42 suites, 19 runnable. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: JailbreakBench, registered as a cost corpus rather than a detection one Reviewed https://jailbreakbench.github.io/ — a standardized benchmark for LLM jailbreaking, MIT, with a live artifacts repository of state-of-the-art adversarial prompts. Measured before judging: of the 100 black-box random-search prompts, 0.300% of characters are non-ASCII, and 99 of 100 carry some. But it is CJK ideographs and accented Latin that a token-level search happened to find useful — not a homoglyph, not an invisible carrier, not a Unicode attack. Scoring detection on it would repeat the category error #743 identified for the GCG suffixes: disarm is structurally blind to an attack written in valid text, and correctly so. So it is wired for the question it can actually answer, which is #743's other half: what does a sanitizer do to traffic it cannot and should not defend against? A guardrail preset in front of an LLM rewrites every request whether or not it recognises anything. subject altered shortened >=1% disarm 100.0% 0.0% decancer 100.0% 0.0% unidecode 99.0% 0.0% stdlib / ftfy 0.0% 0.0% pyunormalize 0.0% 0.0% disarm rewrites all 100 without losing length. That is #743 reproduced on a second, independent corpus, which is the point of having it. Two guards so the numbers cannot be misread, both asserted by tests. The detector column is a census, because flagging valid text carrying no Unicode trick is not a win and missing it is not a failure. And nothing here re-runs a model, so an alteration rate is not a defence rate — the corpus has per-prompt `jailbroken` labels, but this suite cannot use them and says so. Registry is 43 suites, 20 runnable. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * feat: encoding-obfuscation, from arXiv:2508.14070's supporting information Evaluated three papers. One is buildable, one is out of scope, one is on topic but not specified precisely enough to reconstruct without inventing the mapping. **arXiv:2508.14070 — built.** The supporting information carries the 20 base prompts verbatim, so they are quoted rather than invented. Of the attack families the main text names, Base64, hexadecimal and ROT-n are deterministic and reconstructed exactly; leetspeak is not, because its substitution table is a free choice, so it is left out rather than guessed. The full 591-variant corpus stays with `special-char-attack`, still waiting on data. This makes #729 measurable. Textual encoding obfuscation is "neither handled nor named as out of scope, and `detect_encoding` is the name a reader finds first" — so the suite asks whether the boundary sits where the documentation implies. disarm flags 0 of 15 and decodes 0 of 15, which is the correct answer and is now a recorded number rather than an argument. Both columns are censuses and a test asserts they are never directed: a tool that decoded base64 here would be doing something surprising, not something better. `confusable-homoglyphs` flags 100% of pure ASCII, its third indiscriminate result in this registry. **arXiv:2406.18510 (WildTeaming) — out of scope, not added.** Zero mentions of unicode, homoglyph, zero-width, invisible or confusable across the whole paper. It mines in-the-wild natural-language jailbreaks, and a third valid-text jailbreak corpus would duplicate what JailbreakBench already measures. **arXiv:2604.10271 (Doppelgänger Injection) — on topic, deliberately not built.** Homoglyph substitution for adversarial stylometry, by the same author as arXiv:2508.15840 which is already wired. It gives isolated examples (h -> U+04BB, i -> U+0131, e -> U+0435) and substitution rates (5%, 10%, 50%, 100%) but contains no table and no listing, so the mapping would be my choice rather than the paper's. The distinct thing it could add is a recovery-versus- rate curve, which nothing here measures; that is worth building only with the mapping derived from the vendored UTS #39 table and stated as such. Registry is 44 suites, 20 runnable. Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * docs: record what is not a benchmark, and why Evaluated arXiv:1712.06751 (HotFlip). Not added, and it fails on three independent grounds rather than one: - Its alphabet is the CharCNN-LSTM character set — ASCII. No Unicode signal to find, the same reason `jailbreakbench` is registered as a cost corpus. - No released corpus. The only URL in the paper is OpenNMT, the framework. A GitHub search finds third-party reimplementations, all unlicensed and none official. - The attack is white-box and gradient-dependent, so the vectors cannot be derived from the paper either — the first of this batch to fail the derivability test that `mcp-tag-block-concealment` and the others passed. Its contribution already reaches the registry through its descendants: RAG-Pull builds on Boucher's Bad Characters ("Following~\cite{boucher2021badcharacters}, each candidate is a set of insertion operations, bounded by a maximum perturbation count M"), cites HotFlip three times, and both descendants are wired. HotFlip is where character-level adversarial attacks start; Bad Characters is where they become Unicode, and that is the point this harness picks them up. Mining paper lists produces more rejections than additions, so the three tests and the four papers rejected so far are now written down — a paper should not be re-evaluated from scratch, and the reasons are specific enough to revisit if an upstream changes. The tests, in order: is there an artifact (check `arxiv.org/src/`, not just the abstract page and GitHub), is it licensed (read the README, not just the licence field), and is it in scope (measure before judging). Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): emoji-delimiter segmentation, and fix a battery-wide zero Two independent changes, both found by running the harness. leaderboard: discriminations() intersected subjects listwise across every item, so one narrow benchmark capped the shared-subject set for all the others. `weaponizing-unicode` reports `flagged_by_a_detector`, which only the two subjects in the detector role can answer; that put every item below the three-subject floor and produced a battery where every discrimination and therefore every composite was exactly 0.000. Bradley-Terry, fed the same data, was unaffected, which is what made the zeros visible as a defect rather than a result. Switched to pairwise deletion with a mean rest-score (a sum would score a subject answering ten items on a different scale from one answering four) and a MIN_REST_ITEMS floor. Two regression tests, both verified failing on the previous code. academic: new `emoji-delimiter-segmentation`, derived from arXiv:2411.01077 (Emoji Attack, ICML 2025). The first vector in the battery that conceals nothing — a visible emoji inserted inside a word, splitting it for a subword tokenizer. The paper's repository carries no licence in its metadata or its README, so nothing from it is used: the construction is quoted, the emoji set is the UCD's Emoji_Presentation property (1,219 code points) and the carrier is the neutral ASCII pair damage.classify_removal already uses. Scores a five-way ladder over alphanumeric runs: rejoined / split_widened / split_survives / letter_substituted / carrier_destroyed. The last branch exists because without it the empty string has no runs and the delete- everything control scored 100% on a metric describing an emoji that became a word. Cites #757: split_widened is that issue's mechanism seen from the other side. Refs #909 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): score a compiled pipeline, and stop zero-filling detectors The harness scored disarm's presets and profiles and never its ability to compile a pipeline for a purpose — the capability TextPipeline exists for. Adding it exposed two defects in the harness itself. subjects: new `disarm-composed`. Entered as a separate subject rather than another disarm surface, on the precedent that two versions of one tool may compete. The step list is declared once and never varied per benchmark — a composition chosen per suite would be best-of-N with extra steps — and is hashed into the version string, so changing a flag changes the subject key. The composition is the change #910 proposes for llm_guardrail, so what it scores is a proposal already on the table. It is knowingly not equivalent to that profile: per #911, strip_pua is the one ProfileSpec field TextPipeline cannot express, and the gap is left visible rather than papered over. academic: `rows_detected_by_best_detector` and `detected_any` were recorded for every subject, so the eight that claim no DETECT capability scored 0/22,370 rather than being absent. That turned "has a detector at all" into a large z-score advantage for the one subject that has one. It is not a rounding matter: removing the zero-fill moves disarm from first place to second, behind unidecode. The harness's stated rule everywhere else is that a surface a subject lacks is absent, not zero; these two were the exception. academic: `cover_text_intact` and `visible_label_intact` compared the original bytes, so any case-folding configuration scored 0 — the same score as deleting the sentence outright. Both now compare against the cover text put through the same surface, with a non-empty guard so a delete-everything surface cannot pass trivially. Three regression tests; the detector one verified failing on the previous code. Refs #909, #910, #911 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): adopt strip_pua in the declared composition #911 merged as #912, so `strip_pua` is reachable from `TextPipeline` and the composed subject can now express what every screening profile does. Declared explicitly rather than left to the default: it defaults to False, so omitting it would quietly reintroduce the exact divergence #911 was filed about, and nothing in the run output would say so. The step digest moves c896be95 -> 2b92ca62, which is the mechanism working — a changed composition cannot be mistaken for the previous measurement. Verified: `get_pipeline("llm_guardrail")` and the composed pipeline now both strip 137,468 of 137,468 Private Use Area code points. The composite does not move, and that is a gap rather than a null result. No suite in the battery scores the PUA: `weaponizing-unicode` excludes all 1,049 of them from its denominator on purpose (confusability decided by rendering glyphs makes PUA an artefact), and its `rewritten_by_the_sanitizer` is a census with no direction. So the harness found the gap and cannot see the fix. A UCD `General_Category=Co` census would separate the field — measured over a 3,437-point sample, six subjects remove every one, three keep every one, `text-unidecode` is partial — but that is a new suite, not part of adopting this fix. Rebased onto main. Two conflicts resolved: - `adversarial_eval/metrics.py`: #903/#904 landed the same #759 capability on main with a better signature — a dotted path rather than the callable this branch passed, which does not survive being handed to a worker process. Took main's implementation wholesale and adapted the caller. - `tests/test_meta_benchmark.py`: the conflicting assertion sat directly above three whole incoming test functions; all three kept. Refs #909, #911, #912 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): one pipeline per use case, and a UCD Private Use census The composed subject was a single pipeline run against every benchmark, which measured one more fixed configuration and nothing about compiling for a purpose. Composability means a hand-crafted pipeline per use case. A pipeline chosen per *benchmark* would have been best-of-N wearing a different hat: map the cost suites to a light composition and the coverage suites to a heavy one and the mapping does the winning. So there is no mapping. Each use case is a separate subject scored on the whole battery — prompt-hygiene, retrieval-key, review-display — and each meets the suites it is going to lose. The trade-offs are visible as a shape across the report rather than collapsed into one number: review-display corrupts 0% of clean rows and leaves 84% of confusables unreached; retrieval-key reaches the most and corrupts 91%. That immediately paid for itself. prompt-hygiene scored 0% on `mcp-tag-block-concealment` where retrieval-key scored 100%, and the reason is #914: `demojize` is the only composable step that removes the Plane 14 TAG block. So #910's proposed flip of `demojize` on `llm_guardrail` would trade a text-injection primitive for a concealment channel. #910 now carries a STOP warning above its scope; #914 filed for the separation that has to come first. The gap is pinned by a test so it cannot be quietly reversed to improve a score — the gap is the finding. normative: new `ucd-private-use`, so a #911-shaped regression is measurable rather than merely arguable. The direction is Unicode's, not this harness's: of the 137,468 General_Category=Co code points, exactly zero are `Allowed` in UTS #39's IdentifierStatus.txt, and that count is reported as `identifier_allowed` so the claim is recomputable from the page. Separates the field cleanly — six subjects remove every one, three keep every one, text-unidecode is 95.3% with 4.7% substituted, and null-baseline destroys the carrier. Refs #909, #910, #911, #912, #914 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): each benchmark declares its job, and the subject answers with the surface it ships for that job The harness scored disarm through one default surface everywhere, so a confusables benchmark could be answered by a pipeline built to preserve text for a reviewer. That measures the harness's choice, not the library. A user who knows disarm picks the preset that fits the job; scoring it any other way is measuring it badly. I had rejected exactly this as best-of-N in disguise. That was wrong, and the difference is where the choice is declared. Best-of-N picks the winning surface per *measurement*, after seeing scores. Here the job is declared on the BENCHMARK, from what its source deploys into, identically for every subject; one surface answers each benchmark; and that same surface pays that benchmark's cost measurements, so nothing can fold hard for coverage and gently for damage on one suite. `selection_effect_best_of_n` still reports what the choice is worth against the best available surface. Six jobs, 25 benchmarks assigned. disarm answers them with canonicalize, llm_guardrail, rag_ingest, strip_format and code_context — every one a surface the library ships and documents for that purpose. Two mappings were checked against the higher-scoring alternative and kept: * CONFUSABLE_FOLD stays on `canonicalize` though `strip_obfuscation` folds more of confusables.txt onto a shared form (64.1% vs 57.8%). Per #614, `strip_obfuscation` NAMES 49 rows the TR39 table folds, so a spoof and its target stop being equal rather than become equal, and part of that higher score is both sides being named alike. * It is also not `normalize_confusables`, the surface whose name most suggests it: the bare TR39 fold with no normalization reaches 27.1%, less than half of canonicalize, because much of the table is resolved by NFKC. That second point exposed a census gap: `transforms()` enumerated PRESETS and profiles only, so `normalize_confusables` was invisible to every suite in the registry. Now included. Effect: with the right surface per job, `mcp-tag-block-concealment` and `zero-width-stylometry` both go to 100% across every measurement, and disarm's composite interval narrows from [-0.08, 1.08] to [0.21, 0.92] — the only subject whose interval excludes zero. Refs #909, #614 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): peer-relative completeness, pairwise dominance, and an honest overlap claim Rebuilding the plot found three defects, all in the same family as the one fixed in `discriminations()`: a listwise rule letting one narrow benchmark decide the whole result. leaderboard: `Item.complete` judged a subject against every key ANY subject produced. `disarm` answers four directed measurements on the TAG-block suite because it has a detector; a transform-only subject answers two, and was therefore marked incomplete on three benchmarks. That is the zero-fill bug from the previous commit wearing the opposite sign — exclusion instead of a false zero. Completeness is now judged against a subject's peers, meaning the cohort that answered the same key set, so a detector is compared with detectors and a plain transform with transforms. leaderboard: `pareto()` intersected the field across every benchmark, so `weaponizing-unicode` — whose only directed measurement needs a detector, so two subjects answer it — reduced the comparable set below two and returned no frontier at all. Silently: the non-dominated count simply vanished from the report. Dominance is now taken pairwise on the axes both subjects answered, with a minimum shared-axis count so an incomparable pair is not declared non-dominated by default. report: rendering a frontier crashed on an axis a tool never answered rather than printing the absence. Fixed, and the report now states the count out loud: 11 of 12 subjects are non-dominated, so dominance separates almost nothing here. With 13 axes a subject need only lead on one to be safe. leaderboard: the overlap blocker claimed "no two subjects have non-overlapping bootstrap intervals" while `separated_pairs` only ever checked ADJACENT pairs. The claim was false as soon as any non-adjacent pair separated, which `disarm` [0.21, 0.92] against `stdlib` [-1.40, -0.11] does. Adjacency remains the right test for whether an ordering is noise; the message now says so and carries the all-pairs count (1 of 55) beside it. Four regression tests. Plot and reproduction script rebuilt from this run. Refs #909 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): the composite was ranking on eight agreeing axes and calling it thirteen Five defects, all found by Richard reading the published plot. Every figure below was verified before being fixed. 1. Discrimination was fitted over the whole field, controls included. `cronbach_alpha` and `axis_correlations` both exclude them, with a comment explaining that `null-baseline` and `identity` are bad at everything at once and so manufacture agreement between axes that oppose. `discriminations` is the same family of statistic and was the one still counting them. It changed answers: `uts39-confusables` weighed +0.11 with the controls in and 0.00 without. 2. Correcting that exposed the real problem rather than fixing it. Five of thirteen axes now carry zero weight — `corruption-cost` and `uts39-confusables` among them, which are the two axes the published plot was drawn on. A corrected item-total correlation is negative for an axis that opposes the rest, and `max(0, r)` turns that into exclusion, so on an opposed battery the weighting deletes precisely the pole that makes it a trade-off. `corruption-cost` computes to r = -0.68. The composite is an average over the eight axes that agree with each other; it was published as an average over thirteen. Now a blocker, naming the axes. 3. The controls then did their job. With cost at zero weight the composite rewards destruction, and `null-baseline` — which deletes all input — scored above five real libraries. That is now a hard blocker: a subject refusing the job outscoring subjects attempting it voids the aggregate, whatever the intervals say. 4. The z-scale was fitted on every subject, so a library set the units in proportion to how many of its configurations were entered. Four of twelve fitted tools were disarm, because this harness composed three pipelines for disarm and none for anyone else — a third of the scale. Fitted on one subject per library now: nine libraries. The compositions are still scored, they no longer vote on where zero is. 5. `confusable-homoglyphs` answers 4 of 13 axes; the shared-axis floor needs 6; so it met nobody and landed on the frontier untested, while the comment above that floor claimed it prevented exactly that. Incomparable subjects are held out and reported as such. The count is 10 of 11, not 11 of 12. The plot is republished as the audit rather than the ranking, and both of its false self-descriptions are gone: "the page and its data cannot disagree" covered two recomputed figures out of many, and five of the thirteen benchmarks are marked `derived` by the harness's own --list, not externally released. Five regression tests. Refs #909 Signed-off-by: Richard Quinn <quinn.richard@gmail.com> Assisted-by: Claude:claude-opus-5[1m] * bench(meta): a bootstrap draw with no weighted axis is dropped, not scored 0.0 `composite` returns 0.0 when its denominator is zero — a placeholder for "no weighted axis to average", not a measurement. The bootstrap appended it, so a run of exact zeros entered the distribution the quantiles are read from. With five axes already clamped to zero weight, a resample can draw only those. It happened in 10 of 400 draws. That is exactly 2.50%, and the 2.5% quantile reads index 10, so the placeholder did not p…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ICANN's Reference LGR for the Second Level, Latin script (25 October 2024)
defines 25 variant sets over a 231-element repertoire, expanding to 51 blocked or
fallback pairs. 23 carry the Latin Generation Panel's own comment "Glyphs either
homoglyph or nearly identical".
canonicalizecollided 2 of them.All 19 that a single code point can express now collide, and the four hostname
rows in the issue are closed:
This was never a defect in the fold. These are SAME-SCRIPT Latin-to-Latin pairs,
and disarm's confusable data is cross-script — most of the code points are not
TR39 sources at all, so the two sets never met. The rows come from a third data
file with its own admission criterion, beside the cross-script supplement (#342)
and the attested rows (#597).
Both decisions the issue raised are settled, and settling them made this smaller.
THE TARGETS ARE NOT ASCII. Folding
żtozmerges it with the bare letter,which this LGR does not block — the over-collapse
search_keyalready commits at1,534 non-LGR merges — so the target must be the other member of the pair. The
issue proposed folding to the lower code point pairwise. That is not a function
here:
ỉappears in three pairs and would need three targets,ỷin two. And itregresses one row — it makes
əthe SOURCE and overwrites its existing TR39 foldto
e, undoing a fold that already reaches ASCII, in the very pair the issueflags as an inconsistency.
Read as equivalence classes instead: 16 classes over 35 code points, with each
representative taken from the class's existing ASCII fold where one exists and the
lowest code point otherwise. Two classes resolve to ASCII, 14 to a non-ASCII Latin
letter, and the
ǝ/əinconsistency is repaired rather than entrenched.build.rsasserts what makes a non-ASCII target safe rather than trusting it: thevalue must be Latin, and it must not itself be a source. All 2,290 rows were ASCII
before and nothing checked it, which is the weakest state to change an invariant
from. Both conditions verified to fail the build — a Greek target and a chaining
target each produce their own message.
BUILD.RS:216 IS NOT RELAXED. The contraction table is the wrong home for a reason
that decides it before the assert is reached:
contraction::contractis calledonly from
src/hostname.rsbehindif contractions, so it is unreachable fromnormalize_confusablesand puttingn̄→ñthere would not collide the pair —this issue's whole claim. The two multi-code-point rows are dropped at a cost of 2
of 23, and the class they belong to is #836: six Latin bases where tilde and
macron disagree on precomposition, which is where a sequence mechanism should be
designed.
tests/test_lgr_pairs.pyasserts both directions, and the second matters more.The LGR's other 23 pairs are commented "Required for use with Common LGR" —
transitivity artefacts of running it beside the Greek and Cyrillic rulesets — and
its own Variants section says they can be removed when it is used standalone.
Folding
u/üora/áwould strip legitimate diacritics from every languagethat uses them, so those are pinned as MUST NOT collide. That is what stops a
later well-meaning import of the whole variant set.
Latin table 2,273 -> 2,290; the five documented counts and the key fixture move
with it, and
docs/provenance.mdgains the LGR as a source with its version date.Closes #831
Refs #336, #342, #597, #715, #801, #836, #762
Verification
Both decisions from your review are implemented as agreed, with the pairwise-convention defect recorded in the data file's header.
🤖 Generated with Claude Code