Skip to content

Cross-file duplicate Quote rows caused by pre-correction StableId hashing #220

Description

@DutchJaFO

Title

Cross-file duplicate Quote rows caused by pre-correction StableId hashing

Body

While doing a full manual title-consistency review across the bundled quote sources (#181/#217), found an architectural gap that #181's rule/alias mechanisms cannot fix: QuoteIdentity.StableId hashes normalize(quote text) + "|" + normalize(source text) using the raw, pre-correction source text. When the same real-world quote appears in two different bundled files under two different raw source-text spellings, this produces two different quote ids for what is genuinely the same quote — and no per-quote or per-source correction can merge them after the fact, because the id itself was already computed from the "wrong" text before any correction ever runs.

Two confirmed live instances:

  1. The Godfather Part II: NikhilNamal17 quote 3eb18c68-6211-ed48-b54a-6ebbb42d6c79 ("Keep your friends close, but your enemies closer.", raw source "The Godfather II") and vilaboim quote 9ee90d54-cf0e-014c-b4a1-4378ddd2d791 (identical quote text, raw source "The Godfather Part II"). Minimal per-source conflict-resolution rule file + curated field-override preload #181 added a SourceAliasRule so NikhilNamal17's copy now resolves to the correct "The Godfather Part II" Source — but the two rows remain two separate Quote entries, both linked to the same (now-correct) Source, because their ids were already fixed at conversion time from the differing raw source text.
  2. The Shawshank Redemption: NikhilNamal17 quote 7e53658c-0c3a-6546-8c12-c5e4af23c9f8 (raw source "Shawshank Redemption") and another NikhilNamal17 quote e97a8197-657f-ce43-8032-4edd8e3549a3 (raw source "The Shawshank Redemption") have byte-for-byte identical quote text ("Hope is a good thing, maybe the best of things, and no good thing ever dies.") but different ids, for the same reason.

Why this is a different problem from #181's own scope: #181's SourceAliasLookup and ConflictResolutionRule both operate on an already-assigned entity id — they correct which Source a quote links to, or which field value a quote displays, but never merge two genuinely-duplicate Quote rows that already have different ids.

Possible directions for a future issue (none decided, needs its own design):

  • Canonicalize source text (via the same alias mechanism, or a shared normalization step) before QuoteIdentity.StableId is computed, so an id-generation pass run after aliases exist would naturally produce matching ids. This likely only helps for future/regenerated data, not already-shipped ids.
  • A manual Quote-level merge mechanism — an explicit "these two ids are the same quote, keep one, soft-delete the other" operation, usable after the fact regardless of when the ids were assigned.

Not designed or scoped yet. See docs/milestones/data-import-sources/181-minimal-conflict-resolution-rule-file-plan.md for the full context of the review this came out of.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions