cate() + Twins ITE / OBD OPE vs top techniques + the honest meta-lesson#32
Merged
Conversation
…TE/OPE techniques Adds cate() (per-unit CATE / T-learner ITE) to GFormulaBackdoorAgent, and extends the honest real-data suite to two new regimes, benchmarked against the TOP techniques (not strawmen): - Twins individual counterfactuals (examples/causal_mbrl_twins.py): our g-formula T-learner vs the S-learner and X-learner, scored by PEHE against the exact per-pair effect. Honest NULL -- every top ITE method ties a constant at the noise floor (~0.32); real sparse binary individual effects are near-unlearnable (population ATE recoverable, per-unit effect not). DRL can't do counterfactuals; nor can the top causal ITE methods here. - OBD off-policy evaluation (examples/causal_mbrl_obd.py): top OPE estimators (IPS/SNIPS) agree and are unbiased on the clean random logs (unconfounded, sample-limited), 3x biased under a controlled confounding illustration; certificate is a correct true-negative on known-propensity logs. S- and X-learner added to examples/_causal_baselines.py. REAL_DATA.md rewritten around the honest meta-lesson across all four datasets: causal point estimates do NOT reliably beat strong contenders on real data (parity when well-behaved; fragile/noisy/ unlearnable when hard); the defensible edge is the decision+certificate layer, not the number. TDD (cate recovers heterogeneity, averages to the ATE, arm-guards); ruff + pyright(src) clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the honest real-data suite to two new regimes, benchmarked against the top techniques (as
requested — no strawmen), and keeps the genuine OBD results rather than discarding them.
New library capability
GFormulaBackdoorAgent.cate(data)— per-unit CATE (a T-learner ITE estimate). TDD'd (recovers knownheterogeneity, averages to the ATE, guards non-binary / missing arms).
Twins — individual counterfactuals vs the S-/X-learner
Both potential outcomes are known, so the per-pair effect is exact. Honest null: PEHE ties at the
noise floor for everyone — ours (T) 0.322, S-learner 0.320, X-learner 0.318, constant 0.320. Real
sparse binary individual effects are near-unlearnable; DRL can't do counterfactuals, and neither can
the top causal ITE methods here. Population ATE recoverable (−0.025), per-unit effect not.
OBD — off-policy evaluation vs IPS / SNIPS (kept, not discarded)
Honest, non-data-mined findings: (1) canonical task — IPS/SNIPS recover the random policy's value from
BTS logs, in the ballpark of ground truth, sample-limited on the 10k slice; (2) a controlled
confounding illustration where IPS balloons 3×; (3) the certificate as a correct true-negative on
the clean known-propensity logs. Honest scope stated: OBD's decisions are clear-cut, so the
certificate can't show its edge there.
The honest meta-lesson (REAL_DATA.md)
Across all four datasets: causal point estimates do not reliably beat strong contenders — parity
when well-behaved (NHEFS), fragile when hard (LaLonde scatter), unlearnable at the individual level
(Twins), sample-limited/unbiased-then-fooled in OPE (OBD). The defensible edge is the
decision + certificate layer (LaLonde, RCT-vindicated), not the number.
Scripts out-of-CI (network /
causaldata/obp/ sklearn), ruff-clean;pyright srcclean; cate TDD'd.