Repository navigation
Build representative calibration for useful duplicate detection and search #20
Description
Goal
Build compact, reviewable calibration data that makes codedupes duplicate detection and semantic search useful on real code across Python, Rust, C, JavaScript, and TypeScript.
Calibration supports the product. This project is not trying to become a general embedding-model benchmark or a research-grade model calibrator. The useful outcome is simple: relevant duplicates and search results should rank highly, obvious lookalikes should not dominate the output, and the defaults should behave reasonably in every supported language.
Why this work exists
The former polyglot corpus was too synthetic to support trustworthy defaults:
- most units were tiny, mirrored helpers rather than code people maintain;
- some positive labels were behaviorally wrong, unreachable, or incomplete;
- nearby negatives and search probes did not constrain the thresholds well;
- the selection process rewarded recall too aggressively and could return noisy results;
- the evidence was uneven across languages.
The replacement should use behavior-tested code and explicit judgments, with enough variety to expose bad defaults. It should grow because a real coverage or calibration gap was found, not to satisfy arbitrary fixture counts.
Working principles
- Apply the same quality bar to all five supported languages.
- Keep at least five defensible easy and five defensible medium positive pairs per language. Do not remove valid examples merely to balance buckets.
- Hard pairs are optional. Keep one only when it is genuinely difficult in the available search scope; a mate that reliably appears among the leading
semble find-relatedresults usually belongs in an easier bucket. Sparse language fixtures require judgment rather than a rigid rank rule. - A positive means a maintainer could reasonably consolidate the duplicated implementation or extracted region. Behavior equivalence is recorded separately when it matters.
- Test fixture behavior. Review close false positives and search relevance explicitly.
- Optimize selected defaults for F1 with a precision floor. Prefer recall only when the F1 result is tied, because noisy output makes the tool unusable.
- Report results by language and positive difficulty, plus precision, recall, and F1 for duplicate admission, default-visible duplicates, and search.
- Add the smallest fixture or tooling change that resolves a demonstrated gap. Full demo applications, arbitrary quotas, and duplicate variants added only to raise counts are counterproductive.
Phase 1 — development corpus and calibrated defaults
PR #22 implements this phase.
- Replace the legacy synthetic corpora with one versioned manifest and explicit pair/search judgments.
- Use runnable, behavior-tested fixtures for Python, Rust, C, JavaScript, and TypeScript.
- Reach at least five comparable easy and five comparable medium positive pairs in every language without reducing valid existing coverage.
- Review nearby negatives, partial regions, deterministic findings, and every prediction above the selected fixture gates.
- Select duplicate, hybrid-visibility, and search settings with a precision-safe F1 objective.
- Record per-language difficulty recall and pooled precision/recall/F1.
- Reproduce the selections independently and verify CPU/MPS decision parity.
- Publish the corpus contract, methodology, limitations, and machine-readable result.
The phase-one corpus contains 118 annotated units, 78 positive judgments, 178 reviewed negatives, and 60 search probes. It is development data and does not by itself establish performance on unrelated repositories.
Phase 2 — representative validation and final calibration
- Run the phase-one defaults on a small, independent, hand-reviewed sample of real code for each supported language. Review the top duplicate candidates and representative behavioral searches, because that is the output users actually see.
- Record the sample, model, revision, scope, judgments, and observed failure modes so the check is reviewable and repeatable.
- Add targeted positives, close negatives, or probes only when the real-code review exposes a missing common pattern or an unstable threshold decision. Preserve all valid phase-one coverage.
- Recalibrate when the added evidence supports a change, then spot-check the result again on the independent sample.
- Publish the final language/difficulty counts, duplicate and search precision/recall/F1, and any remaining limitations.
Issue #20 is complete when this second phase shows that the shipped defaults produce useful, non-garbage top results in all five languages and the final evidence is documented. The goal is representative confidence, not a claim of universal accuracy.
Explicit non-goals
The following are not required to close this issue unless the phase-two evidence demonstrates a concrete need for one of them:
- fixed quotas such as 60 positives, 100 negatives, or 30 probes per language;
- bootstrap confidence intervals or a statistically representative estimate of the entire open-source ecosystem;
- a bespoke demo project or exhaustive language-feature matrix for every fixture;
- calibration of every embedding model;
- cross-language duplicate calibration;
- class/type-definition embedding, extractor changes, or other analyzer features that deserve separate product issues.
Activity
[-]Rework the polyglot calibration corpus: units, categories, labels, negatives, and selection statistics are not representative[/-][+]Build representative calibration for useful duplicate detection and search[/+]feat: replace and recalibrate the polyglot corpus (phase 1) (#22)
feat!: cap the default check report at 20 pairs in terminal and JSON (#…
pszemraj commentedon Sep 22, 2026
Status update after #23 (merged as 90ae2b1).
Done since phase 1 (#22)
checkreport: terminal and--jsonnow emit the same ≤ 20 primary pairs, actionable tiers (exact,traditional_near,hybrid_confirmed) ranked ahead ofsemantic_high_confidence, then anysemantic_reviewpairs admitted by--include-review.--max-duplicates N|allcontrols the budget;--include-review,--show-all, and--full-tablelift it. The JSON summary gainedactionable_duplicates,reported_actionable_duplicates,truncated_by_tier,strict_unused, andhidden_only_failure.--show-all --jsonorReportPolicy(show_all=True), which are still complete.What the self-scan says about the remaining problem
codedupes check src --jsonon this repo (540 Python units, gte-modernbert on MPS) yields 87 hybrid pairs: 0 actionable, 49semantic_high_confidence, 38semantic_review. The default report is therefore 20 uncorroborated embedding hits, alljaccard: null, ordered by semantic similarity alone. The top of that list is dominated by wrapper/implementation pairs (find_similar_to_query↔_find_similar_to_query_unlocked,resolve_encode_plan↔_resolve_encode_plan,_fingerprint_local_model_dir↔_fingerprint_local_model_dir_cached) and near-identicalcollect_specsoverrides across language backends. Three real extractable duplicates did surface in that list and were fixed in #23, so the tier is not noise, but on real code the density ofsemantic_high_confidenceis high enough that the cap, not the gate, is what keeps the report readable. The cap made the report short; it did not make it precise.What remains (phase 2, unchanged)
src/self-scan above is one Python data point and is not independent.Not touched by #23 and still open:
potentially_unusedand the diagnostics lists are unbounded in JSON; exact-family grouping (#17) becomes more pressing now that a 7-copy family (21 edges) would fill the entire default budget on its own.pszemraj commentedon Sep 22, 2026
Status update from the schema v4 branch (#25: exact families for #17, ranked and bounded unused, exact-first ordering fix, gitignore-aware scans).
The self-scan is no longer one Python data point. All four trees were run with default flags (combined mode, MPS, unsandboxed) on the v4 branch:
codedupes check . --no-gitignore— the whole checkout including the gitignoredlocal-scratch/(dozens of copied ML-experimenttrain_gpt.pyfiles): 4,037 units, 18,676 hybrid edges. 7,642 of those wereexactedges over 175 families / 1,104 units, and the pre-v4 default report showed none of them: it was 20traditional_nearpairs of two-statementMLP/Blockclasses at Jaccard 1.0, because the analyzer sort key let a perfect-score near pair outrank exact pairs whose scores areNone. v4 fixes the ordering and reports families, so the report now leads withmain×5 (479 lines) and the other largest families. A plaincodedupes check .now honours.gitignoreand sees 708 units (src/plusscripts/), 104 findings, 0 actionable: the same wrapper/implementationsemantic_high_confidencelist as thesrcscan.codedupes check tests --no-default-excludes— 1,282 units, 453 findings, 13 actionable (3 exact families + 5traditional_near+ 5hybrid_confirmed), all 13 in the default report, 399semantic_high_confidencemostly truncated. Every exact family isstructural_hash, i.e. tests that share a skeleton and differ only in string literals; the previous scan's "byte-identical test bodies" reading was an artifact of the exact record carrying nomethod.codedupes check test_fixtures— 156 units across all five languages, 60 findings, 3 actionable (the planted cowsaymake_borderspair, now labelledtoken_hash, plus the newexact_familyfixture), 52semantic_high_confidence.codedupes check src— 561 units, 88 findings, 0 actionable, 50semantic_high_confidence/ 38semantic_review, unchanged in substance from the earlier scan.Failure modes to look for in the phase-2 sample, now with a second source of evidence:
resolve_encode_plan↔_resolve_encode_plan,find_similar_to_query↔_find_similar_to_query_unlocked,compute_embeddings↔compute_embeddings_with_identity) and near-identical sibling overrides across backends (collect_specs×4,_statement_count×2) still dominatesrc/'ssemantic_high_confidencelist.traditional_nearat Jaccard 1.0 on two-statement classes (MLP,Block,RMSNormacross experiment copies): identical identifier sets, structurally different bodies, exempt from the tiny filter because classes are. These are weak findings that reach confidence 1.0; whether the tiny filter should see class statement counts, or the near tier should carry a size term, is a calibration question for phase 2.wrap×4 in cowsay, the Caggregate_usage_*trio), which is what the corpus labels expect.Still open: the phase-2 independent sample itself.