Fuzzy dedup silently merges distinct long, near-identical symbols (length-differing pairs, both >= 12 chars)

[Alex-R-A]

Summary

Pass 2 fuzzy dedup in graphify/dedup.py merges two distinct symbols when their normalized labels are long (>= 12 chars) and differ by a small edit, e.g. a trailing plural s, or one name being a strict prefix-extension of the other. The guards that prevent this for short labels (_is_variant_pair, _short_label_blocked) both early-out for labels >= 12 chars, so long near-twins are never protected. The absorbed symbol disappears from graph.json and its edges (callers, imports, etc.) are silently reattached to the surviving node, corrupting call/dependency data.

Minimal repro (uses graphify's own dedup functions)

from rapidfuzz.distance import JaroWinkler
from graphify.dedup import _norm, _is_variant_pair, _short_label_blocked, _MERGE_THRESHOLD

a = _norm("getActiveSession()")     # 'getactivesession'  (16 chars)
b = _norm("getActiveSessions()")    # 'getactivesessions' (17) - a distinct batch/plural function

score = JaroWinkler.normalized_similarity(a, b) * 100
print("JW:", round(score, 2))                          # 98.82
print("MERGE_THRESHOLD:", _MERGE_THRESHOLD)            # 92.0  -> over threshold, merges
print("is_variant_pair blocks?:", _is_variant_pair(a, b))            # False (guard is < 12 chars only)
print("short_label_blocked?:",   _short_label_blocked(a, b, score))  # False (guard is < 12 chars only)

score (98.82) >= 92 and neither guard applies (both return early at max(len(a), len(b)) >= 12), so the pair is unioned in Pass 2. The same-file partition check (the if norm_label == neighbor_norm: branch) only fires when the normalized labels are identical, so it does not catch this either, even when both symbols are defined in the same file.

Effect

Two genuinely different functions in the same file, e.g. a single-item function and its batch sibling whose names differ only by a trailing s, collapse into one node. The plural is absent from graph.json; queries for it return nothing, and the singular's callers/dependents wrongly include the plural's call sites. The merge is silent, only an aggregate Deduplicated N node(s) (... M fuzzy) line is printed, so an unknown fraction of those M fuzzy merges may be false positives of this kind.

Expected

Distinct symbols whose normalized names differ in length, especially where one is a strict prefix-extension of the other (getActiveSession is a prefix of getActiveSessions; parseConfig vs parseConfigFile), should not be merged. Such pairs are almost never duplicates.

Suggested fix

Either:

• Apply the length-difference guard to all label lengths, not only < 12 (drop the if max(len(a), len(b)) >= 12: return False early-out in _short_label_blocked); or

• Add a targeted check in Pass 2 before the union, skip the merge when one normalized label is a strict prefix-extension of the other:

  lo, hi = sorted((norm_label, neighbor_norm), key=len)
  if hi != lo and hi.startswith(lo):
      continue

  Prefix-extension pairs are essentially never duplicates, so this is low-risk.

Workaround status

There appears to be no config knob to disable or tune fuzzy dedup. --dedup-llm only adjudicates pairs scoring in [75, 92); pairs like this score ~98 and are auto-merged in Pass 2 before the LLM tiebreaker runs, so it does not help. Affected users currently have no workaround short of patching dedup.py.

Environment

• graphifyy 0.8.35

[Alex-R-A]

Confirmed on real codebases, with a root-cause check and a working fix.

Root cause is definitely Pass 2 fuzzy in deduplicate_entities (ruled out the alternatives):

• _make_id does not collide for the two names, they get distinct ids.
• Pass 1 exact-normalization keys differ (getactivesession vs getactivesessions), no merge.
• build.py:deduplicate_by_label keys on _norm_label, also different, no merge.

So Pass 2 fuzzy is the only step that merges them: JaroWinkler ~98 >= _MERGE_THRESHOLD (92), and both _is_variant_pair / _short_label_blocked early-out at >= 12 chars, so long pairs are unguarded.

Severity is higher than it looks. On two real-world JS / TS codebases, adding the guard below dropped the reported "fuzzy" merge count by roughly half on one and roughly 80% on the other, i.e. the majority of fuzzy merges were distinct symbols (singular/plural functions, base/specialized names) being silently collapsed, with the absent symbol's edges (callers/imports) reattached to the survivor. Before the fix, a singular/plural function pair defined in the same file produced only one node and the plural's call sites were wrongly attributed to the singular; after, both nodes exist and callers split correctly.

The fix (surgical, blocks only strict prefix-extension pairs, placed in Pass 2 right after the existing short-label guards):

# in deduplicate_entities, Pass 2, immediately after:
#   if _is_variant_pair(norm_label, neighbor_norm): continue
#   if _short_label_blocked(norm_label, neighbor_norm, score): continue

# Block a fuzzy merge when one normalized label is a strict prefix-extension of the
# other (e.g. "getactivesession" vs "getactivesessions", "parseconfig" vs
# "parseconfigfile"). The short-label guards above skip labels >= 12 chars, so long
# singular/plural and base/specialized pairs would otherwise collapse into one node.
# Identical-norm pairs are unaffected (the `_hi != _lo` test), so the same-label
# cross-file handling further down still runs.
_lo, _hi = sorted((norm_label, neighbor_norm), key=len)
if _hi != _lo and _hi.startswith(_lo):
    continue

Prefix-extension pairs are essentially never duplicates, so this is low-risk for code graphs. (For doc/semantic graphs it could in principle keep an abbreviation and its expansion separate, e.g. config vs configuration, which seems acceptable.)

Minimal repro on current main (graphify's own functions, synthetic names, no paths):

from rapidfuzz.distance import JaroWinkler
from graphify.dedup import _norm, _short_label_blocked, _MERGE_THRESHOLD

a, b = _norm("getActiveSession()"), _norm("getActiveSessions()")
s = JaroWinkler.normalized_similarity(a, b) * 100
print(round(s, 2), ">=", _MERGE_THRESHOLD, "| short_label_blocked:", _short_label_blocked(a, b, s))
# 98.82 >= 92.0 | short_label_blocked: False   -> merged in Pass 2

Happy to open a PR with the guard if that's useful.

[safishamsi]

Fixed in e477825. Added a prefix-extension guard in both Pass 2 and _llm_tiebreak in dedup.py: if one normalised label is a strict prefix of the other (getactivesession / getactivesessions), the merge is skipped regardless of JW score. Option (a) — dropping the >=12 early-out from _short_label_blocked — was rejected: it breaks legitimate typo merges like GraphExtractor/GraphExtractar. Two new regression tests added.