You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cross-file duplicate Quote rows caused by pre-correction StableId hashing #220
Cross-file duplicate Quote rows caused by pre-correction StableId hashing
Body
While doing a full manual title-consistency review across the bundled quote sources (#181/#217), found an architectural gap that #181's rule/alias mechanisms cannot fix: QuoteIdentity.StableId hashes normalize(quote text) + "|" + normalize(source text) using the raw, pre-correction source text. When the same real-world quote appears in two different bundled files under two different raw source-text spellings, this produces two different quote ids for what is genuinely the same quote — and no per-quote or per-source correction can merge them after the fact, because the id itself was already computed from the "wrong" text before any correction ever runs.
Two confirmed live instances:
The Godfather Part II: NikhilNamal17 quote 3eb18c68-6211-ed48-b54a-6ebbb42d6c79 ("Keep your friends close, but your enemies closer.", raw source "The Godfather II") and vilaboim quote 9ee90d54-cf0e-014c-b4a1-4378ddd2d791 (identical quote text, raw source "The Godfather Part II"). Minimal per-source conflict-resolution rule file + curated field-override preload #181 added a SourceAliasRule so NikhilNamal17's copy now resolves to the correct "The Godfather Part II" Source — but the two rows remain two separate Quote entries, both linked to the same (now-correct) Source, because their ids were already fixed at conversion time from the differing raw source text.
The Shawshank Redemption: NikhilNamal17 quote 7e53658c-0c3a-6546-8c12-c5e4af23c9f8 (raw source "Shawshank Redemption") and another NikhilNamal17 quote e97a8197-657f-ce43-8032-4edd8e3549a3 (raw source "The Shawshank Redemption") have byte-for-byte identical quote text ("Hope is a good thing, maybe the best of things, and no good thing ever dies.") but different ids, for the same reason.
Why this is a different problem from #181's own scope:#181's SourceAliasLookup and ConflictResolutionRule both operate on an already-assigned entity id — they correct which Source a quote links to, or which field value a quote displays, but never merge two genuinely-duplicate Quote rows that already have different ids.
Possible directions for a future issue (none decided, needs its own design):
Canonicalize source text (via the same alias mechanism, or a shared normalization step) beforeQuoteIdentity.StableId is computed, so an id-generation pass run after aliases exist would naturally produce matching ids. This likely only helps for future/regenerated data, not already-shipped ids.
A manual Quote-level merge mechanism — an explicit "these two ids are the same quote, keep one, soft-delete the other" operation, usable after the fact regardless of when the ids were assigned.
Not designed or scoped yet. See docs/milestones/data-import-sources/181-minimal-conflict-resolution-rule-file-plan.md for the full context of the review this came out of.
Title
Cross-file duplicate Quote rows caused by pre-correction StableId hashing
Body
While doing a full manual title-consistency review across the bundled quote sources (#181/#217), found an architectural gap that #181's rule/alias mechanisms cannot fix:
QuoteIdentity.StableIdhashesnormalize(quote text) + "|" + normalize(source text)using the raw, pre-correction source text. When the same real-world quote appears in two different bundled files under two different raw source-text spellings, this produces two different quote ids for what is genuinely the same quote — and no per-quote or per-source correction can merge them after the fact, because the id itself was already computed from the "wrong" text before any correction ever runs.Two confirmed live instances:
3eb18c68-6211-ed48-b54a-6ebbb42d6c79("Keep your friends close, but your enemies closer.", raw source "The Godfather II") and vilaboim quote9ee90d54-cf0e-014c-b4a1-4378ddd2d791(identical quote text, raw source "The Godfather Part II"). Minimal per-source conflict-resolution rule file + curated field-override preload #181 added aSourceAliasRuleso NikhilNamal17's copy now resolves to the correct "The Godfather Part II" Source — but the two rows remain two separate Quote entries, both linked to the same (now-correct) Source, because their ids were already fixed at conversion time from the differing raw source text.7e53658c-0c3a-6546-8c12-c5e4af23c9f8(raw source "Shawshank Redemption") and another NikhilNamal17 quotee97a8197-657f-ce43-8032-4edd8e3549a3(raw source "The Shawshank Redemption") have byte-for-byte identical quote text ("Hope is a good thing, maybe the best of things, and no good thing ever dies.") but different ids, for the same reason.Why this is a different problem from #181's own scope: #181's
SourceAliasLookupandConflictResolutionRuleboth operate on an already-assigned entity id — they correct which Source a quote links to, or which field value a quote displays, but never merge two genuinely-duplicate Quote rows that already have different ids.Possible directions for a future issue (none decided, needs its own design):
QuoteIdentity.StableIdis computed, so an id-generation pass run after aliases exist would naturally produce matching ids. This likely only helps for future/regenerated data, not already-shipped ids.Not designed or scoped yet. See
docs/milestones/data-import-sources/181-minimal-conflict-resolution-rule-file-plan.mdfor the full context of the review this came out of.