Flagged during Sprint 2 (# FIXME(flagged) in trace_analysis/thrustAnalysis.py, exact_mc_perm_test).
The observed statistic uses the median difference, but the permuted statistic uses the mean difference:
diff = np.abs(np.median(xs) - np.median(ys)) # observed: median
...
k += diff < np.abs(np.mean(zs[:n]) - np.mean(zs[n:])) # permuted: mean
Comparing a median-based observed value against a mean-based null is internally inconsistent and yields an invalid p-value.
Action needed: decide on a single statistic (mean or median) for both, or — per the lab standard — replace this with reRandomStats (FisherResamplingTest).
Flagged during Sprint 2 (
# FIXME(flagged)intrace_analysis/thrustAnalysis.py,exact_mc_perm_test).The observed statistic uses the median difference, but the permuted statistic uses the mean difference:
Comparing a median-based observed value against a mean-based null is internally inconsistent and yields an invalid p-value.
Action needed: decide on a single statistic (mean or median) for both, or — per the lab standard — replace this with
reRandomStats(FisherResamplingTest).