feat: on_empty, so absence and a stripped value stop sharing a key (#728)

[raeq]

Summary

Every preset and key builder maps some non-empty input to "", so a value that was entirely stripped is indistinguishable from one that was never there. sanitize_filename was the only surface that guarded it, with the _ sentinel from #485.

Re-measured over the 292,531 assigned code points (unchanged by #937):

surface	→ ""	excluding PUA
slugify	249,175	111,707
strip_obfuscation	140,251	2,783
search_key	139,921	2,453
catalog_key	139,918	2,450
sort_key	138,404	936
canonicalize / _strict / skeleton_key	137,955	487
ml_normalize	4,110	4,110
sanitize_filename	0	0

A caller storing search_key(username) as a uniqueness key has 2,453 non-PUA code points, every string built from them, and "no username" competing for one slot.

This is not the homoglyph collision the key builders exist to produce — аdmin and admin should meet. It collapses absence onto a value, which arXiv:2608.06508v1 §2.2 calls code-side semantic collapse and §7.5 says is fixed by moving the sentinel outside the value range.

on_empty applies only when the input was non-empty

That is the whole point, and my first cut got it wrong: it substituted for an empty input too, which put absence and a stripped value straight back into one slot — the collision this exists to break. Measuring caught it before it shipped.

search_key("", on_empty=NUL)         # ''   — absence keeps its own key
search_key("​", on_empty=NUL)   # NUL  — stripped to nothing

The three stripped values still share a key, and that is correct: they are all input that reduced to nothing, one fact rather than three. What changes is that absence is no longer one of them.

The sentinel is the caller's to choose and disarm cannot check it — search_key("​", on_empty="admin") equals search_key("admin"). Documented and asserted, rather than pretended away.

Item 4: the census is frozen

Nine docstrings now carry a number, and a number nobody re-measures stops being right the first time a strip class widens. tests/test_empty_key.py measures all ten surfaces over the full assigned codespace and fails with a message naming what to update.

Scope

Python-only for now, as digit_policy was in #885. It is a post-pass on the output — the answer is a property of the result, not of any transform — and the Rust and binding half belongs with #896 rather than beside it, where it would collide with that work.

Two house rules I tripped

.. warning:: renders as those exact characters on the site, so the note is bold prose like has_anomalies'. And the test's invisible literals are named escapes (#802).

Verification

• cargo fmt --check, clippy -D warnings, cargo doc --no-deps, perf_lint.sh — clean
• pytest -n 8 --dist loadfile — 7,179 passed, 42 skipped (31 new)
• mypy, ruff, mkdocs build --strict, run_doc_tests.py 41/41 — clean

Closes #728

[github-actions]

📄 Docs preview: https://a79800df.disarm-docs.pages.dev

[Copilot]

🟡 Changes recommended

A few newly added docs/tests contain inaccurate or misleading wording (notably around the “docstrings claim” census test and sentinel guarantees) that should be corrected before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a Python-only on_empty option to the key-builder APIs so an absent value ("" input) no longer shares the same key as a non-empty input that strips to empty, addressing the “semantic collapse” reported in #728. This keeps current defaults unchanged while letting callers opt into a sentinel for “stripped-to-nothing” outputs, and it freezes the documented “empty output” counts via a Unicode census test.

Changes:

• Add on_empty: str | None to search_key, catalog_key, sort_key, and skeleton_key, applied as a post-pass only when the input was non-empty.
• Document the empty-output case (with frozen counts) across the relevant docstrings and limitations docs.
• Add a full assigned-codepoint sweep test that asserts the frozen “empty key” census.

File summaries

File	Description
tests/test_empty_key.py	New census + behavioral tests for empty-key surfaces and on_empty semantics.
tests/test_api_stability.py	Updates expected parameter lists to include on_empty on the four key builders.
python/disarm/_presets.py	Introduces _EMPTY_KEY_CENSUS, _on_empty() helper, and threads on_empty into key builders; adds empty-output docstring notes.
python/disarm/_api.py	Adds empty-output documentation note to slugify.
docs/limitations.md	Documents the empty-key pitfall, shows on_empty usage, and adds a warning about sentinel choice.
CHANGELOG.md	Changelog entry describing on_empty and the frozen census.

Review details

Suppressed comments (3)

tests/test_empty_key.py:37

• The comment says the sentinel is something no builder can produce, but these builders will produce it when on_empty=SENTINEL is passed (by design). Reword this to make it clear the claim is only about the default output space (without on_empty), or avoid asserting an unverified property in a comment.

#: A sentinel no builder can produce: it is stripped by all of them.
SENTINEL = "␀"  # SYMBOL FOR NULL

tests/test_empty_key.py:74

• The failure message says "docstring and census say" but the test does not read the docstrings, so this can mislead when debugging failures. Consider updating the message to reference the census only (it can still instruct updating docstrings as a follow-up action).

    assert (total, nonpua) == (total_want, nonpua_want), (
        f"{name}: measured {total:,}/{nonpua:,}, docstring and census say "
        f"{total_want:,}/{nonpua_want:,}. If this is a deliberate change, update "
        "_EMPTY_KEY_CENSUS and the docstring in the same commit."

docs/limitations.md:643

• Grammar: this line is missing a word and reads awkwardly ("A value a real input also keys to …").

!!! warning "The sentinel is yours to choose, and disarm cannot check it"
    A value a real input also keys to reintroduces the collision one step over:
    `search_key("\u200b", on_empty="admin")` equals `search_key("admin")`. Pick something

• Files reviewed: 6/6 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[raeq]

CI caught a real one: the census was measured on a Unicode 16.0.0 host and CI's 3.12 is 15.1.0, so the three surfaces that reach "" through transliteration were high by exactly the 16.0 additions — 51 on search_key, 63 on ml_normalize. Same skew #815 hit, same fix.

Pinned at 15.1.0 (the version CI runs) with the pin as a named constant. Re-measured under a 15.1.0 interpreter: all ten surfaces match CI's numbers exactly. The gate compares exactly on the pin and asserts a lower bound on any newer host — a newer UCD only adds code points, so a shrinking count still fails — so neither branch is a skip.

The docstring-parse gate from the last round earned its keep immediately: my first re-numbering pass gave docstrings each other's numbers, and it caught that before push.

[raeq]

The gate refused a second time, as designed, and the message named the reason: CI's 3.12 ships Unicode 15.0.0, and I had pinned at 15.1.0 from a 3.13 host on the assumption they matched.

Installed 3.12 and measured rather than assumed. Nine surfaces are identical across the two versions; slugify is 627 lower at 15.0.0 — every 15.1 addition it drops. Assuming equality would have failed the gate a third time on that surface alone.

Pinned at 15.0.0, the oldest any CI interpreter runs, with the constant's comment listing what each supported CPython ships. Exact branch verified under 3.12, every surface matching.