Tags: jturner-uofl/pycorpdiff
Tags
feat(lexical): Hill numbers + rarefaction for size-fair diversity (0.… …1.0a33) pcd.lexical.hill_numbers + pcd.lexical.rarefaction -- ecology diversity (Hill 1973; Hurlbert 1971; Chao et al. 2014) for size-fair comparison of different-sized corpora. hdd is rarefaction at size 42; this generalises it. 7 tests (659 total).
feat(sense_drift): three story-carrying drift visualizations (0.1.0a32) plot_composition() (stacked-area sense share over time -- the takeover), plot_decline() (slopegraph early->late, coloured by verdict), and an enriched plot() with the calibrated threshold rule + permutation p-value. 652 tests; ruff+mypy clean.
feat(sense_drift): fall-off hunt — decline/obsolescence detection (0.… …1.0a31) decline_report() + sense_trajectories() on SenseDriftResult: classify each reference sense rising/stable/declining, split declines into obsolescence (absolute count falls) vs dilution (share falls, count holds). On CBD-PubMed both declines are dilution (bile-duct doubled in absolute output). 651 tests.
fix(sense_drift): permutation-null-calibrated flagging + p-value (0.1… ….0a30) A label-shuffle control showed the in-sample control chart over-flags (out-of-sample periods look novel vs a reference fit on themselves). sense_drift(n_permutations=N) now calibrates the flag threshold from the label-shuffle null and reports SenseDriftResult.p_value. Default unchanged. On CBD-PubMed: genuine drift 2012-2024 at permutation p=0.024. 648 tests.
feat(semantic): sense_drift — explainable LSCD-grounded sense-drift d… …etection (0.1.0a29) Detect *and explain* when a corpus's sense distribution drifts from a reference period. Mahalanobis novelty-to-nearest-sense (Lee et al. 2018) + margin-density/JSD control chart with sustained-run confirmation (Sethi & Kantardzic 2017; Rabanser et al. 2019), over per-period sense clusters (Giulianelli et al. 2020; Montariol et al. 2021). Explanation layer classifies change type (emergence / frequency_shift / broadening) and surfaces log-ratio distinctive terms. 14 tests; ruff + mypy clean.
0.1.0a28: embedding-based word-sense induction (induce_senses) + clas…
…sifier audit
New `pycorpdiff.senses` module — the package's first word-sense
induction surface, built to defend a hand-built sense classifier with
an unsupervised second opinion rather than trusting the buckets on
faith.
Public API:
- induce_senses(items, embeddings, ...) -> SenseInductionResult.
Bring-your-own embeddings (never embeds internally); deterministic
clustering (seeded k-means default, agglomerative option; no
UMAP/HDBSCAN) so the audit is byte-stable; k auto-selected by
silhouette or passed explicitly. Document- and token-in-context
(unit="token") granularity.
- SenseInductionResult.agreement_with(reference_labels) -> ARI,
V-measure, contingency vs a hand-built labelling.
- SenseInductionResult.leakage_audit(reference_labels, k) -> records
whose embedding geometry disputes their reference label
(deterministic, targeted alternative to a random spot-check).
- SenseInductionResult.share_over_time(freq) -> computed
sense-fraction trajectory.
scikit-learn (already in the [semantic] extra) is required and lazily
imported, so importing pycorpdiff never pulls it.
Tests: 18 new (deterministic labels, agreement recovery, planted-leak
detection, token-mode aggregation, edge cases). Tutorial gains §8b
showcasing the audit workflow on a reproducible three-sense corpus.
Docs: design.md Result matrix, README cheat-sheet + layer surface,
CHANGELOG.