feat: on_empty, so absence and a stripped value stop sharing a key (#728)

CHANGELOG.md

@@ -72,6 +72,38 @@ compatibility (see [RELEASING.md](RELEASING.md)).
 
   Rust gets `api::is_mixed_script_per_word` and `api::has_bidi_conflict_per_word`; the
   existing functions are unchanged.
+- **`on_empty` on the four key builders, and the empty-key census in nine docstrings
+  (#728).** Every preset and key builder maps some non-empty input to `""`, so a value
+  that was entirely stripped is indistinguishable from one that was never there.
+  `sanitize_filename` was the only surface that guarded it, with the `_` sentinel from
+  #485.
+
+  Measured at Unicode 15.0.0, the oldest version CI runs: `search_key` takes **139,870**
+  single characters to `""` (2,402 excluding the PUA), `slugify` 243,399, `canonicalize`
+  137,955.
+  The version is stated because the census is not one number — a 16.0.0 host counts the
+  16.0 additions each surface takes to `""`, and the first version of the gate, measured
+  on one, failed CI by exactly those. The gate now compares exactly on the pinned version
+  and asserts a lower bound on any newer one, so neither branch is a skip.
+  A caller storing `search_key(username)` as a uniqueness key has all of them, every
+  string built from them, **and** "no username" competing for one slot.
+
+  This is not the homoglyph collision the key builders exist to produce — `аdmin` and
+  `admin` *should* meet. It collapses absence onto a value, which arXiv:2608.06508v1 §2.2
+  calls code-side semantic collapse and §7.5 says is fixed by moving the sentinel outside
+  the value range rather than by normalizing harder.
+
+  `on_empty` applies **only when the input was non-empty**, which is the whole point:
+  substituting for an empty input too would put absence and a stripped value back in one
+  slot. `search_key("")` is `""` with or without it; `search_key("\u200b")` is the
+  sentinel. The three stripped values still share a key, and that is correct — they are
+  all *input that reduced to nothing*.
+
+  The census is frozen by `tests/test_empty_key.py`, so a strip class that widens becomes
+  a diff somebody reads rather than a silent change to nine documented numbers.
+
+  Python-only for now, as `digit_policy` was in #885: it is a post-pass on the output, and
+  the Rust and binding half belongs with #896 rather than beside it.
 
 - **`decode_smuggled(text)` and the `smuggled` anomaly kind — decode what a hidden run
   *spells* (#701).** disarm strips the three ASCII-smuggling carriers and, since #700,

docs/limitations.md

@@ -587,6 +587,65 @@ The key builders are excluded from all of this. `search_key`, `catalog_key` and
 two spellings collide (see [`find_key_collisions`](api/predicates.md#find_key_collisions)).
 They are documented as keys, not as cleaners.
 
+### A key can be the empty string, and absence is not a value
+
+Every preset and key builder maps some non-empty input to `""`. A value that was entirely
+stripped is then indistinguishable from a value that was never there.
+
+| surface | single characters → `""` | excluding PUA |
+|---|---|---|
+| `slugify` | 243,399 | 105,931 |
+| `strip_obfuscation` | 140,200 | 2,732 |
+| `search_key` | 139,870 | 2,402 |
+| `catalog_key` | 139,867 | 2,399 |
+| `sort_key` | 138,404 | 936 |
+| `canonicalize` / `canonicalize_strict` / `skeleton_key` | 137,955 | 487 |
+| `ml_normalize` | 4,047 | 4,047 |
+| **`sanitize_filename`** | **0** | **0** — returns `_` |
+
+Measured at **Unicode 15.0.0** — the oldest version CI runs — over every assigned code point, and
+frozen by `tests/test_empty_key.py` so a strip class that widens shows up as a diff. The
+version matters: a 16.0.0 host assigns more code points, and the surfaces that reach `""`
+through transliteration count them; `canonicalize`, `sort_key` and `skeleton_key` do not.
+
+This is **not** the homoglyph collision the key builders exist to produce. Those are
+deliberate — `аdmin` and `admin` *should* meet. This one collapses **absence** onto **a
+value**:
+
+```python
+from disarm import find_key_collisions, search_key
+
+groups = find_key_collisions(
+    ["admin", "", "\u200b", "\u0301\u0302", "\u00ad", "bob"], key="search_key"
+)
+empty = [g for g in groups if g.key == ""][0]
+assert set(empty.values) == {"", "\u200b", "\u0301\u0302", "\u00ad"}
+```
+
+A caller storing `search_key(username)` as a uniqueness key has 2,402 non-PUA code
+points, every string built from them, **and** "no username" competing for one slot. First
+writer takes it; everyone after collides with a record that is not a user.
+
+The four key builders take `on_empty` for it — the fix `sanitize_filename` already made
+with `_`. It applies only when the *input* was non-empty, so absence keeps its own key:
+
+```python
+NUL = "\u2400"  # SYMBOL FOR NULL — a value you have checked none of your inputs keys to
+
+assert search_key("", on_empty=NUL) == ""  # absence
+assert search_key("\u200b", on_empty=NUL) == NUL  # stripped to nothing
+```
+
+The three stripped values still share a key, which is right — they are all *input that
+reduced to nothing*, one fact rather than three. What changes is that absence is no longer
+one of them.
+
+!!! warning "The sentinel is yours to choose, and disarm cannot check it"
+    A value a real input also keys to reintroduces the collision one step over:
+    `search_key("\u200b", on_empty="admin")` equals `search_key("admin")`. Pick a value
+    you have verified none of your inputs keys to — disarm knows the table, not your data.
+
+
 ### `strip_accents` deletes Indic vowel signs
 
 A Latin acute and a Devanagari vowel sign are both general category `Mn`. In Latin an

python/disarm/_api.py

@@ -640,6 +640,16 @@ def slugify(
         'n-a'
         >>> slugify("🔥", default="N/A")  # default is sanitized, not returned raw
         'n-a'
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **243,399** single
+    characters reduce to ``""`` here (105,931 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    There is no ``on_empty`` here: this returns text rather than a key. The
+    four key builders take one.
     """
     _sw = stopwords if isinstance(stopwords, (tuple, list)) else list(stopwords)
     _rp = replacements if isinstance(replacements, (tuple, list)) else list(replacements)

python/disarm/_presets.py

@@ -41,6 +41,70 @@
 _NON_DEFAULT_DIGIT_POLICIES = ("tr39", "preserve")
 
 
+#: The Unicode version the census below was measured under — the OLDEST any interpreter
+#: in CI runs (CPython 3.12 ships 15.0.0; 3.13 ships 15.1.0; 3.14 ships 16.0.0). `tests/test_empty_key.py` compares exactly on that
+#: version and asserts a lower bound on any newer one.
+#:
+#: The version has to be stated, because "assigned code points" is not one set. The first
+#: version of this table was measured on a 16.0.0 host and was high by exactly the 16.0
+#: additions each surface takes to `""` — 51 on `search_key`, 63 on `ml_normalize` — and
+#: CI could not see them and failed the gate. The second pin, 15.1.0 from a 3.13 host,
+#: was refused by CI's 3.12 at 15.0.0: nine surfaces are identical across the two and
+#: `slugify` is 627 lower, which is every 15.1 addition it drops. Measured, not assumed. Surfaces that reach `""` only by
+#: stripping (`canonicalize`, `sort_key`, `skeleton_key`) are identical across the two;
+#: the ones that reach it through transliteration and the confusable fold are not.
+_EMPTY_KEY_CENSUS_UCD = "15.0.0"
+
+#: Single-character inputs whose key is `""`, as (all assigned, excluding the PUA), at
+#: `_EMPTY_KEY_CENSUS_UCD`. Frozen by `tests/test_empty_key.py`.
+_EMPTY_KEY_CENSUS = {
+    "canonicalize": (137_955, 487),
+    "canonicalize_strict": (137_955, 487),
+    "strip_obfuscation": (140_200, 2_732),
+    "ml_normalize": (4_047, 4_047),
+    "search_key": (139_870, 2_402),
+    "catalog_key": (139_867, 2_399),
+    "sort_key": (138_404, 936),
+    "skeleton_key": (137_955, 487),
+    "slugify": (243_399, 105_931),
+    "sanitize_filename": (0, 0),
+}
+
+
+def _on_empty(text: str, key: str, on_empty: str | None) -> str:
+    """Apply the caller's empty-key policy (#728).
+
+    Every preset and key builder maps some non-empty input to `""`, so a value that was
+    entirely stripped is indistinguishable from a value that was never there. Measured,
+    `search_key` alone takes 2,453 non-PUA code points there, and every string built from
+    them. A caller storing the key as a uniqueness constraint has all of them, plus
+    "no value", competing for one slot: first writer takes it, everyone after collides
+    with a record that is not a user.
+
+    `sanitize_filename` is the one surface that already reserved a sentinel — `_`, from
+    #485 — and arXiv:2608.06508v1 §7.5 is explicit that the remedy is to redefine the
+    mapping so the sentinel sits outside the value range, not to normalize harder.
+
+    A post-pass rather than a step inside the pipeline, and deliberately so: the answer is
+    a property of the *output*, not of any transform, and the pipeline has no more
+    information at the end than the caller does. It is here for discoverability and to pin
+    what it means, the way `_apply_digit_policy` is (#885).
+
+    **It applies only when the input was not empty**, which is the whole point. An empty
+    input has an empty key legitimately — nothing in, nothing out — and substituting a
+    sentinel there would put absence and a stripped value back in one slot, which is the
+    collision this exists to break. `search_key("")` is `""` with or without the
+    parameter; `search_key("\u200b")` is the sentinel.
+
+    **The sentinel is the caller's to choose, and disarm cannot check it.** A value that a
+    real input also keys to reintroduces the collision one step over; `on_empty` is only
+    safe when it is outside the range the builder can produce.
+    """
+    if key or on_empty is None or not text:
+        return key
+    return on_empty
+
+
 def _apply_digit_policy(text: str, digit_policy: str) -> str:
     """Fold digit variants under *digit_policy* before a key builder runs (#885).
 
@@ -177,6 +241,16 @@ def canonicalize(text: str, *, digit_policy: str = "numeric") -> str:
         gives output byte-identical to not passing it, so no stored key moves. Pass
         ``"tr39"`` when your inputs are Latin identifiers and the extra reach is worth it.
         Do not pass it to text that may carry Arabic, Persian, Indic or Thai numerals.
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **137,955** single
+    characters reduce to ``""`` here (487 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    There is no ``on_empty`` here: this returns text rather than a key. The
+    four key builders take one.
     """
     return _canonicalize(_apply_digit_policy(text, digit_policy))
 
@@ -262,6 +336,16 @@ def ml_normalize(
         'muenchen'
         >>> ml_normalize("José Martínez", fold_case=False)
         'Jose Martinez'
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **4,047** single
+    characters reduce to ``""`` here (4,047 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    There is no ``on_empty`` here: this returns text rather than a key. The
+    four key builders take one.
     """
     return _ml_normalize(text, lang=lang, emoji_style=emoji, fold_case=fold_case)
 
@@ -272,6 +356,7 @@ def catalog_key(
     lang: str | None = None,
     strict_iso9: bool = False,
     digit_policy: str = "numeric",
+    on_empty: str | None = None,
 ) -> str:
     """Library catalog key generation pipeline.
 
@@ -337,8 +422,23 @@ def catalog_key(
         the default and a genuine no-op; ``"tr39"`` reaches more spoofs but destroys the
         numeric reading of Arabic, Persian, Indic and Thai digits. See `canonicalize` for
         the measurements and the trade (#885).
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **139,867** single
+    characters reduce to ``""`` here (2,399 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    ``on_empty`` reserves a sentinel for that case — the fix
+    `sanitize_filename` already made with ``_`` (#485). It applies only when
+    the *input* was non-empty, so absence keeps its own key.
     """
-    return _catalog_key(_apply_digit_policy(text, digit_policy), lang=lang, strict_iso9=strict_iso9)
+    return _on_empty(
+        text,
+        _catalog_key(_apply_digit_policy(text, digit_policy), lang=lang, strict_iso9=strict_iso9),
+        on_empty,
+    )
 
 
 def strip_format(text: str) -> str:
@@ -394,6 +494,7 @@ def search_key(
     *,
     lang: str | None = None,
     digit_policy: str = "numeric",
+    on_empty: str | None = None,
 ) -> str:
     """Search index key generation pipeline.
 
@@ -443,11 +544,24 @@ def search_key(
         the default and a genuine no-op; ``"tr39"`` reaches more spoofs but destroys the
         numeric reading of Arabic, Persian, Indic and Thai digits. See `canonicalize` for
         the measurements and the trade (#885).
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **139,870** single
+    characters reduce to ``""`` here (2,402 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    ``on_empty`` reserves a sentinel for that case — the fix
+    `sanitize_filename` already made with ``_`` (#485). It applies only when
+    the *input* was non-empty, so absence keeps its own key.
     """
-    return _search_key(_apply_digit_policy(text, digit_policy), lang=lang)
+    return _on_empty(
+        text, _search_key(_apply_digit_policy(text, digit_policy), lang=lang), on_empty
+    )
 
 
-def skeleton_key(text: str, *, digit_policy: str = "numeric") -> str:
+def skeleton_key(text: str, *, digit_policy: str = "numeric", on_empty: str | None = None) -> str:
     """A spoof key: the TR39 skeleton plus the prototype classes disarm keeps apart.
 
     Pipeline: NFKC → strip_bidi → strip invisibles → confusables → **prototype
@@ -507,15 +621,27 @@ def skeleton_key(text: str, *, digit_policy: str = "numeric") -> str:
     ``v1.0.1``, ``vI.O.I`` and ``vl.o.l``. For a spoof detector that is the point;
     for a deduplication key over anything carrying a part number, a version or an
     ISBN it destroys the field.
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **137,955** single
+    characters reduce to ``""`` here (487 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    ``on_empty`` reserves a sentinel for that case — the fix
+    `sanitize_filename` already made with ``_`` (#485). It applies only when
+    the *input* was non-empty, so absence keeps its own key.
     """
-    return _skeleton_key(text, digit_policy=digit_policy)
+    return _on_empty(text, _skeleton_key(text, digit_policy=digit_policy), on_empty)
 
 
 def sort_key(
     text: str,
     *,
     lang: str | None = None,
     digit_policy: str = "numeric",
+    on_empty: str | None = None,
 ) -> str:
     """Sort key generation pipeline.
 
@@ -578,8 +704,19 @@ def sort_key(
         the default and a genuine no-op; ``"tr39"`` reaches more spoofs but destroys the
         numeric reading of Arabic, Persian, Indic and Thai digits. See `canonicalize` for
         the measurements and the trade (#885).
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **138,404** single
+    characters reduce to ``""`` here (936 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    ``on_empty`` reserves a sentinel for that case — the fix
+    `sanitize_filename` already made with ``_`` (#485). It applies only when
+    the *input* was non-empty, so absence keeps its own key.
     """
-    return _sort_key(_apply_digit_policy(text, digit_policy), lang=lang)
+    return _on_empty(text, _sort_key(_apply_digit_policy(text, digit_policy), lang=lang), on_empty)
 
 
 def strip_bidi(text: str) -> str:
@@ -735,6 +872,16 @@ def canonicalize_strict(text: str, *, digit_policy: str = "numeric") -> str:
         the default and a genuine no-op; ``"tr39"`` reaches more spoofs but destroys the
         numeric reading of Arabic, Persian, Indic and Thai digits. See `canonicalize` for
         the measurements and the trade (#885).
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **137,955** single
+    characters reduce to ``""`` here (487 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    There is no ``on_empty`` here: this returns text rather than a key. The
+    four key builders take one.
     """
     return _canonicalize_strict(_apply_digit_policy(text, digit_policy))
 
@@ -827,6 +974,16 @@ def strip_obfuscation(text: str, *, digit_policy: str = "numeric") -> str:
         the default and a genuine no-op; ``"tr39"`` reaches more spoofs but destroys the
         numeric reading of Arabic, Persian, Indic and Thai digits. See `canonicalize` for
         the measurements and the trade (#885).
+
+    **The output can be the empty string (#728).**
+
+    Measured at Unicode 15.0.0, **140,200** single
+    characters reduce to ``""`` here (2,732 excluding the Private Use
+    Area), and so does every string built from them. A caller keying a table
+    on this has all of them, plus "no value", competing for one slot.
+
+    There is no ``on_empty`` here: this returns text rather than a key. The
+    four key builders take one.
     """
     return _strip_obfuscation(_apply_digit_policy(text, digit_policy))
 

tests/test_api_stability.py

@@ -388,11 +388,11 @@ def _param_kinds(fn) -> dict[str, str]:
     # `"numeric"` is the default and a byte-identical no-op, so no stored key moves.
     "canonicalize": ["text", "digit_policy"],
     "ml_normalize": ["text", "lang", "emoji", "fold_case"],
-    "catalog_key": ["text", "lang", "strict_iso9", "digit_policy"],
+    "catalog_key": ["text", "lang", "strict_iso9", "digit_policy", "on_empty"],
     "strip_format": ["text"],
-    "search_key": ["text", "lang", "digit_policy"],
-    "sort_key": ["text", "lang", "digit_policy"],
-    "skeleton_key": ["text", "digit_policy"],
+    "search_key": ["text", "lang", "digit_policy", "on_empty"],
+    "sort_key": ["text", "lang", "digit_policy", "on_empty"],
+    "skeleton_key": ["text", "digit_policy", "on_empty"],
     "strip_bidi": ["text"],
     "strip_tags": ["text"],
     "strip_variation_selectors": ["text"],