fix(engine): entity-name hygiene at resolution intake — whitespace normalization and tag-shaped category-label skip - #3277
Open
JoshFunnell wants to merge 2 commits into
Conversation
Production banks were measured (2026-08-08) with entity canonical_name values containing embedded newlines -- extraction artifacts -- which shears any line-oriented consumer (psql -A output, logs, exports). Add _normalize_entity_name() and apply it at candidate-entity intake in _prepare_entities_for_resolution(), collapsing all internal whitespace runs (including \n, \r, \t) to a single space and stripping ends. Case handling is unchanged (the registry matches on LOWER(canonical_name) separately). Tested by tests/test_entity_name_hygiene.py.
Measured (2026-08-08): extraction minted entities named domain:lens, domain:host, domain:memory -- category labels, not entities. They behave like broad tags and measurably poisoned entity-based scoping (61.5% vs 94.0% precision on one subject). Add a conservative tag-shape regex (_is_tag_shaped_name) and apply it in _prepare_entities_for_resolution(), after whitespace normalization and against a lowercased copy of the name, silently skipping matches (logged at debug level). Configured entity labels are deliberately "key:value" shaped (vectorize-ioGH-1558, e.g. tag-type label values like use:use-001), so _prepare_entities_for_resolution() now also accepts entity_labels and exempts any name that resolves as a configured label from the skip -- without this exemption, test_intrabatch_dedup_leaves_label_values_separate and test_entity_resolution_does_not_merge_distinct_label_values regressed. Tested by tests/test_entity_name_hygiene.py (attacking cases for URLs, Windows paths, times, and configured labels come first, per the false-positive risk).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small data-quality fixes at the entity-resolution intake choke point, each motivated by a measured production defect.
Fixes #3275. Fixes #3276.
1. Whitespace normalization of candidate entity names
Candidate names now have internal whitespace runs (including
\n,\r,\t) collapsed to a single space and ends stripped, before matching or creation. Production banks were measured (2026-08-08) storingcanonical_namevalues with embedded newlines — extraction artifacts that shear any line-oriented consumer (psql -A, logs, exports). Case handling is deliberately untouched (the registry already matches onLOWER(canonical_name)).Existing rows are not migrated by this PR: renormalizing stored names can collide with the
(bank_id, LOWER(canonical_name))uniqueness, so cleanup of old rows is a merge operation and left to operators/a follow-up.2. Skip tag-shaped category labels at intake
Extraction was measured minting category labels as entities (
domain:lens,domain:host,domain:memory). They behave like broad tags and measurably poison entity-based workflows: an entity-scoped retrieval selector that pickeddomain:lensplateaued at 61.5% precision; excluding category-shaped names, 94.0% on the same corpus.Names matching a conservative shape —
^[a-z][a-z0-9_-]{1,15}:[a-z][a-z0-9_-]{1,24}$after normalization, checked on a lowercased copy — are skipped with a debug log and a counter line. The shape deliberately does not match URLs (//), Windows paths, times (digits before the colon), or names containing spaces/dots.Design note discovered in testing: configured entity labels are deliberately
key:value-shaped (e.g.use:use-001), so they match the same regex. They are exempted via the existing label lookup (is_label_entity), keeping the exact-match label path intact. Consequence: an unwanted artifact that happens to collide with a configured label key survives the skip — that seemed the right trade, and it is test-pinned.Where the fixes live, and why there
_prepare_entities_for_resolution()inhindsight-api-slim/hindsight_api/engine/retain/link_utils.py— the single choke point both resolution entry paths funnel through, and before the flat-list/entity_to_unitmapping is derived, so the resolver's positional invariant (output index-aligned with input) is untouched. Dropping names deeper inside the resolver would have broken that alignment.Tests
hindsight-api-slim/tests/test_entity_name_hygiene.py, 27 tests, non-matching names asserted first (URLs, Windows paths, times,re: subject, plain names, configured labels). Regression across the entity suites: 208 passed; the small set of pre-existing failures onmain(a Turkish-İ lowercasing case and label-extraction tests needing an API key) was verified unchanged in count and identity before/after this diff.