Normalize embedding inputs with rehuman (code_safe) before semantic comparison #24

Problem

Semantic inputs are embedded as the unit's raw source. Comments, docstrings, and string literals carry typography that has nothing to do with what the code does — curly quotes, em dashes, ellipsis characters, zero-width and other invisible characters, emoji in log messages and docstrings — and two implementations that differ only in that noise get slightly different vectors. The deterministic fingerprints are mostly immune (the structural hash collapses every string literal to <STR> and both hashes drop comments), so this is a semantic-similarity concern, not an exact-duplicate one.

Proposal

Normalize the prepared embedding input with rehuman (Rust core, prebuilt wheels for Linux/macOS/Windows, releases the GIL, ~1k docs/s/core) using the code_safe preset:

import rehuman

OPTIONS = rehuman.Options.code_safe_preset().replace(
    normalize_other=True,        # fold ellipsis and friends to ASCII
    unicode_normalization="nfc", # canonical form, no visual change
)

code_safe keeps non-Latin scripts (the default clean() deletes them) and is the preset intended for code-heavy corpora. Apply it to the text handed to the embedding backend only; never to fingerprints, identifiers, CodeUnit.source, or anything a report prints, so byte offsets and rendered snippets stay exact.

Why this is its own change

This is a change to inference inputs, so it has to be treated like one (AGENTS.md, "Caches and calibration evidence"):

• The prepared-input transform is part of embedding identity. Adding normalization means a new cache key space; it must not silently reuse vectors embedded from raw source.
• The phase-1 calibration in test_fixtures/calibration/calibration-results.json was captured on raw source. Recapture on CPU fp32 through the existing scripts, verify MPS decision parity, and report pair-score drift and duplicate/search threshold flips for both built-in models before any default moves.
• Measure whether it changes what users see: rerun the repository scans (., src, test_fixtures, tests --no-default-excludes) and diff the default reports; the interesting question is whether any top-20 semantic_high_confidence result changes, which folds naturally into the Build representative calibration for useful duplicate detection and search #20 phase-2 independent-sample review.
• New runtime dependency: pin per pyproject.toml rules, confirm wheel coverage for the supported Python range, and decide whether platforms without a wheel fall back to raw input (with a diagnostic) or fail.

Acceptance

• Embedding inputs are normalized with the code_safe preset; fingerprints, identifiers, source ranges, and rendered snippets are byte-identical to today.
• Cache identity distinguishes normalized from raw inputs; a warm cache from before the change is not reused for normalized inputs.
• Calibration artifacts regenerated through the existing scripts, with the drift and threshold-flip numbers in the PR.
• Repository-scan diffs for the four trees in the PR, with any changed top-20 semantic results called out.