Five surfaces rewrite `|`, `"` and `` ` ``, and only a Rust comment says so — while limitations.md enumerates the ASCII rows that are *not* applied

[raeq]

Summary

Five surfaces rewrite three printable ASCII characters, and no user-facing page says so. Measured on 0.14.1 across U+0021–U+007E:

surface	ASCII rewrites
canonicalize	|→l, "→'', `→'
canonicalize_strict	same three
strip_obfuscation	same three
normalize_confusables	same three
catalog_key	same three, plus the 26 case folds
search_key, sort_key, ml_normalize	case folds only — no confusable step

The rows are faithful TR39 (confusables_to_latin.tsv:2-4, all three in confusables_upstream_sources.tsv). The finding is not that they are wrong; it is that they are invisible to a reader and that the exclusion policy around them is applied to two thirds of one class.

Where it is written down today — three internal places, none of them a doc:

• src/confusables.rs:232 — "the latin table maps ASCII source code points (U+007C |→l, U+0022 "→'', U+0060 `→'), so ASCII input is not identity even for target="latin"."
• tests/exhaustive_confusables.rs:21-24 — the same sentence, as a justification for keeping ASCII in the sweep.
• Nowhere else. Grepped docs/, python/disarm/, and every docstring on the five surfaces: zero mentions of U+007C, U+0022, U+0060, "vertical line", or ASCII-not-identity.

What a reader is told instead. docs/limitations.md:225-227 enumerates the ASCII rows disarm deliberately does not apply:

"the set includes five ASCII characters (%, 0, 1, I, m) because TR39 is a skeleton transform whose m→rn, I/1→l and 0→O rows disarm deliberately does not apply — folding a legitimate m to rn corrupts prose."

That passage names the exclusions and never mentions the inclusions, so it reads as "ASCII passes through". It does not.

The class is folded asymmetrically

|, I and 1 are three ASCII members of one TR39 confusable class with prototype l. Two are excluded by the policy above; one is not:

>>> disarm.canonicalize("|"), disarm.canonicalize("I"), disarm.canonicalize("1")
('l', 'I', '1')
>>> disarm.is_confusable("|"), disarm.is_confusable("I"), disarm.is_confusable("1")
(True, False, False)

So paypa|.com collapses onto paypal.com and paypaI.com does not. Whichever behaviour is right, one class should not be half-folded, and unmapped_confusables() does not report | as an exposure because it is mapped.

Composition canonicity

A flat-string normalizer has no notion of a field boundary, so the fold reaches the separator. | is a common composite-key separator (DynamoDB sort keys, Redis keys, tenant-scoped identifiers, log lines), and it is the one that folds:

>>> disarm.canonicalize("user|admin")
'userladmin'
>>> disarm.canonicalize("userladmin")
'userladmin'                                # a two-field key and a one-field key collide

canonicalize is documented as "For cleaning untrusted input before comparison, this is the entry point" and "scoped to identifiers", which is the shape a caller composes. If the join happens before the call, the decomposition is destroyed and a value containing a literal l can forge a boundary. /, :, ., - and _ all survive; only | does not, which is exactly the case a caller would not predict.

This is the composition-canonicity level of arXiv:2608.06508v1 §7.3 — a composite object is canonical only if the composition is uniquely decomposable and each field value is canonical. disarm gets the second and silently breaks the first.

Scope

1. Document the three applied ASCII rows next to the five excluded ones in docs/limitations.md, and in the docstrings of canonicalize, canonicalize_strict, strip_obfuscation, normalize_confusables and catalog_key. One sentence: these five surfaces are not identity on printable ASCII.
2. Decide the | / I / 1 asymmetry deliberately and record the reasoning wherever it lands. Either extend the docs/limitations.md:226 exclusion to U+007C — consistent with I/1, and it restores the separator — or keep the fold and say why this member of the class differs.
3. Say in the docs that these surfaces are not composition-safe: canonicalize each field, then join; do not join, then canonicalize. A worked two-field example belongs in docs/user-guide/.
4. If §2 keeps the fold, unmapped_confusables() is not the whole exposure surface for ASCII — a companion listing the ASCII rows that are applied would let a caller check either direction.

Refs #703, #593, #587

Reviewed against arXiv:2608.06508v1 §7.3 (composition canonicity vs value canonicity).

[raeq]

Two additions from measuring these three folds against an external adversarial corpus,
both of which bear on how this resolves.

A fourth place the reader is told the opposite

The issue lists three internal mentions and docs/limitations.md:225-227 as the passage
that reads as "ASCII passes through". THREAT_MODEL.md → Out of scope → Injection
attacks says it outright:

Pure-ASCII payloads such as <script>alert(1)</script> or ' OR 1=1 -- pass through
every transform unchanged (every Unicode transform is a no-op on ASCII).

Both named examples do pass unchanged, so the sentence is true of its own evidence and
false as a general claim. It matters more here than in limitations.md, because it is the
argument for disarm not being an injection defence, and the metacharacters it is wrong
about are injection metacharacters:

>>> disarm.canonicalize("' OR 1=1 --")          # as documented
"' OR 1=1 --"
>>> disarm.canonicalize('admin" OR "1"="1')     # not
"admin'' OR ''1''=''1"
>>> disarm.canonicalize("cat /etc/passwd | mail me")
'cat /etc/passwd l mail me'

The | row is currently load-bearing, by accident

Measured over the 1,683 GCG adversarial suffixes released with
arXiv:2506.12880v2
(MatanBT/gcg-evaluated-data, MIT), every one of which is pure ASCII:

delta	occurrences
" → ''	608
whitespace collapse	543
| → l	139
` → '	32

911 of 1,683 suffixes (54.1%) are rewritten by canonicalize; per split, 50.9% gemma2,
61.6% qwen2.5, 72.4% llama3.1. So the folds are not a rare edge — they reach the majority
of a real ASCII adversarial corpus.

More consequentially, U+007C → l is the only thing in disarm that breaks a
chat-template delimiter spelled in user text. <|im_end|> becomes <lim_endl>, which
tokenizes as prose rather than as a special token (verified against Qwen2.5-1.5B-Instruct's
released tokenizer). Delimiters without a | — Gemma's <end_of_turn>, Llama 2's
[INST] — pass through whole. Filed separately as #742, with the family-by-family table.

I am not arguing the fold should stay on that basis; an accidental control that covers four
model families and misses two is not a control. But if this resolves by dropping the |
row to fix the composite-key collision, <|im_end|> starts passing through every preset,
and that should be a chosen outcome rather than a discovered one.

Refs #742, #743