Skip to content

Every preset and key builder maps non-empty input to "", so absence and a stripped value share one key — sanitize_filename is the only surface that reserves a sentinel #728

Description

@raeq

Summary

Every preset and key builder maps some non-empty input to "", so a value that was entirely stripped is indistinguishable from a value that was never there. sanitize_filename is the only surface in the library that guards it.

Measured on 0.14.1 over all 287,346 assigned code points — single-character inputs whose output is the empty string:

surface assigned code points → "" excluding PUA dominant categories
slugify 244,026 106,558 Co, Lo, So, Mn, Sm
strip_obfuscation 141,526 4,058 Co, Mn, So, Mc, Cf
canonicalize 137,943 475 Co, Mn, Cf, Cc, Zs
canonicalize_strict 137,943 475 same
ml_normalize 4,035 4,035 Mn, So, Mc, Cf, Cc
search_key 2,293 2,293 Mn, Mc, So, Cc, Cf
catalog_key 2,290 2,290 Mn, Mc, So, Cc, Cf
sort_key 570 570 Mn, So, Cc, Mc, Cf
sanitize_filename 0 0 returns _
>>> disarm.search_key(""), disarm.search_key("​"), disarm.search_key("́̂")
('', '', '')
>>> disarm.sanitize_filename("​")     # the one surface with a fallback sentinel
'_'

find_key_collisions does report it, which is the right behaviour and is what makes the shape visible:

>>> disarm.find_key_collisions(["admin", "", "​", "́̂", "­", "bob"], key="search_key")
[KeyCollision(key="", values=["", "\u{200b}", "\u{301}\u{302}", "\u{ad}"], indices=[1, 2, 3, 4])]

Why this is its own shape

This is not the homoglyph collision the key builders exist to produce. Those are deliberate: аdmin and admin should meet. This one collapses absence onto a value, and the two are not the same kind of thing — a caller who stores search_key(username) as a uniqueness key has 2,293 distinct single code points, and every string built from them, occupying one slot that is also the slot for "no username". First writer takes it; everyone after collides with a record that is not a user.

arXiv:2608.06508v1 §2.2 calls this direction code-side semantic collapse and works it out on the Nomad bridge, where bytes32(0) meant both "no message" and "confirmed". §7.5 is explicit that upstream normalization cannot fix it — the remedy is to redefine the mapping so the sentinel is outside the value range. sanitize_filename already did exactly that with _, and #485 is the issue that put it there.

The docstrings do not mention it. search_key, catalog_key and sort_key each carry a Warning about homoglyph collisions and a Stability note, and neither says the key can come back empty. canonicalize says nothing either.

Scope

  1. Document the empty-output case in the docstrings of canonicalize, canonicalize_strict, strip_obfuscation, ml_normalize, search_key, catalog_key, sort_key and slugify, with the per-surface counts above. A caller keying a table needs to know the key can be "" before they find out from a collision.
  2. Give the key builders a way to say so. An on_empty parameter ("" as today / raise / a caller-supplied sentinel) puts the choice where the policy lives, the way find_key_collisions leaves the collision policy to the caller. sanitize_filename's _ is the precedent.
  3. Note it in docs/limitations.md beside the existing collision material, and in whichever docs/user-guide/ page shows a key builder feeding a uniqueness constraint.
  4. Freeze the counts. A tier-3 sweep asserting the per-surface totals turns a silent widening — a future strip class taking more code points to "" — into a diff somebody reads.

Evidence, as contiguous code point ranges per surface: https://gist.github.com/raeq/ef8caa91cbd3425f6f203d3d4ec25817
Harness: https://gist.github.com/raeq/eeb76344aa74e5f0a47a8f4b80889887

Refs #485, #620

Reviewed against arXiv:2608.06508v1 §2.2 and §7.5 (code-side semantic collapse; sentinel separation).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestsecuritySecurity finding

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions