"Normalized" mutual information divides nats by bits and drifts with sample size #212

compute_predictor_correlations documents a "normalized mutual information matrix" that is "comparable to correlation coefficients" (predictor_analysis.py:72), and hardcodes the diagonal to 1.0 (:154-155). The values are not normalized: mutual information in nats is divided by an entropy in bits, and for a continuous column that entropy is a plug-in discrete estimate that reduces to log2(n_distinct) — a measure of sample size, not information.

Reproduction

Two identical columns, where the correct normalized value is exactly 1.0:

import numpy as np, pandas as pd
from microimpute import compute_predictor_correlations

x = np.random.default_rng(3).normal(size=1000)
df = pd.DataFrame({"a": x, "b": x.copy()})
print(compute_predictor_correlations(df, ["a", "b"], method="mutual_info")["mutual_info"].loc["a", "b"])

n=200    0.5285
n=1000   0.5671
n=5000   0.5909

Perfectly dependent variables score around 0.55, and the value drifts upward with sample size. At n=1000: MI is 5.651 nats while _compute_entropy returns 9.9658 bits, which is exactly log2(1000).

Two errors compound. Dividing by H·ln2 to fix the units gives 0.818, still not 1.0, because the log2(n_distinct) proxy is not the entropy of a continuous variable. Because genuinely categorical columns get a real entropy while continuous ones get log2(n), values are not comparable across variable types — and predictor_target_mi (:221-225) is used to rank predictors for imputation.

Locations

• predictor_analysis.py:163-171 — predictor-predictor normalisation
• predictor_analysis.py:221-225 — predictor-target normalisation
• _compute_mutual_information at :529-550 — sklearn, returns nats
• _compute_entropy at :553-568 — log2, returns bits

Suggested fix

Convert units consistently, and use a normaliser valid for continuous variables — a proper NMI estimator, or normalising by min(H(X), H(Y)) computed on the same discretisation used for the MI estimate. Failing that, drop the normalisation, document the units as raw nats, and stop hardcoding the diagonal to 1.0, which currently hides the defect from inspection.

Found during a pre-JOSS-submission audit (#201), reproduced independently by a second reviewer.