Summary
Every preset and key builder maps some non-empty input to "", so a value that was entirely stripped is indistinguishable from a value that was never there. sanitize_filename is the only surface in the library that guards it.
Measured on 0.14.1 over all 287,346 assigned code points — single-character inputs whose output is the empty string:
| surface |
assigned code points → "" |
excluding PUA |
dominant categories |
slugify |
244,026 |
106,558 |
Co, Lo, So, Mn, Sm |
strip_obfuscation |
141,526 |
4,058 |
Co, Mn, So, Mc, Cf |
canonicalize |
137,943 |
475 |
Co, Mn, Cf, Cc, Zs |
canonicalize_strict |
137,943 |
475 |
same |
ml_normalize |
4,035 |
4,035 |
Mn, So, Mc, Cf, Cc |
search_key |
2,293 |
2,293 |
Mn, Mc, So, Cc, Cf |
catalog_key |
2,290 |
2,290 |
Mn, Mc, So, Cc, Cf |
sort_key |
570 |
570 |
Mn, So, Cc, Mc, Cf |
sanitize_filename |
0 |
0 |
returns _ |
>>> disarm.search_key(""), disarm.search_key(""), disarm.search_key("́̂")
('', '', '')
>>> disarm.sanitize_filename("") # the one surface with a fallback sentinel
'_'
find_key_collisions does report it, which is the right behaviour and is what makes the shape visible:
>>> disarm.find_key_collisions(["admin", "", "", "́̂", "", "bob"], key="search_key")
[KeyCollision(key="", values=["", "\u{200b}", "\u{301}\u{302}", "\u{ad}"], indices=[1, 2, 3, 4])]
Why this is its own shape
This is not the homoglyph collision the key builders exist to produce. Those are deliberate: аdmin and admin should meet. This one collapses absence onto a value, and the two are not the same kind of thing — a caller who stores search_key(username) as a uniqueness key has 2,293 distinct single code points, and every string built from them, occupying one slot that is also the slot for "no username". First writer takes it; everyone after collides with a record that is not a user.
arXiv:2608.06508v1 §2.2 calls this direction code-side semantic collapse and works it out on the Nomad bridge, where bytes32(0) meant both "no message" and "confirmed". §7.5 is explicit that upstream normalization cannot fix it — the remedy is to redefine the mapping so the sentinel is outside the value range. sanitize_filename already did exactly that with _, and #485 is the issue that put it there.
The docstrings do not mention it. search_key, catalog_key and sort_key each carry a Warning about homoglyph collisions and a Stability note, and neither says the key can come back empty. canonicalize says nothing either.
Scope
- Document the empty-output case in the docstrings of
canonicalize, canonicalize_strict, strip_obfuscation, ml_normalize, search_key, catalog_key, sort_key and slugify, with the per-surface counts above. A caller keying a table needs to know the key can be "" before they find out from a collision.
- Give the key builders a way to say so. An
on_empty parameter ("" as today / raise / a caller-supplied sentinel) puts the choice where the policy lives, the way find_key_collisions leaves the collision policy to the caller. sanitize_filename's _ is the precedent.
- Note it in
docs/limitations.md beside the existing collision material, and in whichever docs/user-guide/ page shows a key builder feeding a uniqueness constraint.
- Freeze the counts. A tier-3 sweep asserting the per-surface totals turns a silent widening — a future strip class taking more code points to
"" — into a diff somebody reads.
Evidence, as contiguous code point ranges per surface: https://gist.github.com/raeq/ef8caa91cbd3425f6f203d3d4ec25817
Harness: https://gist.github.com/raeq/eeb76344aa74e5f0a47a8f4b80889887
Refs #485, #620
Reviewed against arXiv:2608.06508v1 §2.2 and §7.5 (code-side semantic collapse; sentinel separation).
Summary
Every preset and key builder maps some non-empty input to
"", so a value that was entirely stripped is indistinguishable from a value that was never there.sanitize_filenameis the only surface in the library that guards it.Measured on 0.14.1 over all 287,346 assigned code points — single-character inputs whose output is the empty string:
""slugifystrip_obfuscationcanonicalizecanonicalize_strictml_normalizesearch_keycatalog_keysort_keysanitize_filename_find_key_collisionsdoes report it, which is the right behaviour and is what makes the shape visible:Why this is its own shape
This is not the homoglyph collision the key builders exist to produce. Those are deliberate:
аdminandadminshould meet. This one collapses absence onto a value, and the two are not the same kind of thing — a caller who storessearch_key(username)as a uniqueness key has 2,293 distinct single code points, and every string built from them, occupying one slot that is also the slot for "no username". First writer takes it; everyone after collides with a record that is not a user.arXiv:2608.06508v1 §2.2 calls this direction code-side semantic collapse and works it out on the Nomad bridge, where
bytes32(0)meant both "no message" and "confirmed". §7.5 is explicit that upstream normalization cannot fix it — the remedy is to redefine the mapping so the sentinel is outside the value range.sanitize_filenamealready did exactly that with_, and #485 is the issue that put it there.The docstrings do not mention it.
search_key,catalog_keyandsort_keyeach carry a Warning about homoglyph collisions and a Stability note, and neither says the key can come back empty.canonicalizesays nothing either.Scope
canonicalize,canonicalize_strict,strip_obfuscation,ml_normalize,search_key,catalog_key,sort_keyandslugify, with the per-surface counts above. A caller keying a table needs to know the key can be""before they find out from a collision.on_emptyparameter (""as today / raise / a caller-supplied sentinel) puts the choice where the policy lives, the wayfind_key_collisionsleaves the collision policy to the caller.sanitize_filename's_is the precedent.docs/limitations.mdbeside the existing collision material, and in whicheverdocs/user-guide/page shows a key builder feeding a uniqueness constraint.""— into a diff somebody reads.Evidence, as contiguous code point ranges per surface: https://gist.github.com/raeq/ef8caa91cbd3425f6f203d3d4ec25817
Harness: https://gist.github.com/raeq/eeb76344aa74e5f0a47a8f4b80889887
Refs #485, #620
Reviewed against arXiv:2608.06508v1 §2.2 and §7.5 (code-side semantic collapse; sentinel separation).