Repository navigation
[W08] Rebuild paper failure figures from real artifacts (currently unlabeled synthetic demo) #76
Description
Task metadata
| Field | Value |
|---|---|
| Category | Writing |
| Priority | Medium |
| Complexity | Low |
| Est. days | 1 |
| Related | #68 (W07), PR #74 |
Description
The case-report PDF under paper/build/ uses failure-framework figures (taxonomy, complementarity) whose caption says "representative seed" — but the only artifacts that existed when it was built came from scripts/run_failure_analysis.py --demo, i.e. planted synthetic data (n=150). Nothing in the caption says synthetic, so the figures read as real-cohort results.
Since PR #74 the harness has a --manifest route and real artifacts exist:
python scripts/run_failure_analysis.py \
--manifest data/processed/echo_hubert_manifest.parquet --out results/failure
Real numbers on the 245-row test frame (ridge harness, seed-fixed): echo wins per-example 245/0, LOO marginal value echo 107.3 vs ecg 6.5, drop_echo induces 188 critical failures at silent rate 1.0, drop_ecg 59 at 0.27. The real story is stronger than the synthetic one — dropping echo fails silently across the board.
The .tex source is not in this repo (only paper/build/ outputs), so this needs whoever holds the paper source.
Steps:
- Regenerate the taxonomy / complementarity / dropout figures from
results/failure(real) and swap them into the paper. - Update captions with real n and provenance; anywhere synthetic demo output remains, label it as harness validation on synthetic data.
- Rebuild the PDF so
paper/build/stops carrying figures over unlabeled synthetic numbers.
Acceptance criteria:
- Paper figures trace to the real-manifest run, or are explicitly labeled synthetic.
- No results table or figure in the repo presents demo output as cohort results ([W07] Restate reported numbers from the canonical run #68 acceptance item).
Activity
Name the model behind every failure-analysis artifact
duckyquang commentedon Sep 9, 2026
The real figures exist now. I regenerated the paper's full figure set from the committed sanitized artifacts of the canonical run — no data access needed, everything traces to checkpoint
7cae4f92…/ manifest81694c9b…:failure_report.fused.json— the fused checkpoint's own predictions, not the ridge harness, so the figures finally match Table 4's modelmissing_modality.metrics.json(with the bootstrap CIs)fairness_metrics.jsonAll as PDF (fonttype 42, LaTeX-ready) + 300-dpi PNG.
The fused numbers behind the three views (245-row test fold):
drop_echoinduces 174 critical failures — 135 loud / 39 silent (silent rate 0.22) at +10.1 MAE;drop_ecginduces just 7, but 6 of the 7 are silent (rate 0.86) at +0.8 MAE. Per-example winners: echo 191 / ecg 54. Note this reads differently from the ridge-harness numbers quoted in this issue's description (188 critical, silent rate 1.0) — the fused story is the one the paper should carry, and it lines up cleanly with the below-chanceecg_droppedAUROC: few failures, no observable signal, inverted gate underneath.@sebasmos the files are ready to swap into the
.tex— I'll get them to you today (happy to attach here or push under version control if we'd rather have them tracked). Captions need: fused checkpoint, real test fold, n=245.