Spun out from #1221 as suggested.
add_relative_skill() returns only relative-skill columns: it drops pval and adj_pval before returning. But it calls through to get_pairwise_comparisons() with the default test_type = "non_parametric", so wilcox.test() runs for every model pair (choose(n, 2) per metric per group) and the results are then thrown away.
For callers that only want relative skill (which is what add_relative_skill() is for), those tests are pure overhead. On the FluSight benchmark in #1221 that is ~16.4 s (2.4% of the eval-data build), all discarded. It also produces "cannot compute exact p-value with ties" warnings whenever the scores tie.
Proposal: default add_relative_skill(test_type = NULL), or otherwise let relative-skill-only callers skip the test. Since add_relative_skill() discards the p-values regardless, this changes no output, only removing the wasted computation and the warnings.
get_pairwise_comparisons() itself legitimately returns p-values, so its default is fine; this is specifically about add_relative_skill() computing then discarding them.
Spun out from #1221 as suggested.
add_relative_skill()returns only relative-skill columns: it dropspvalandadj_pvalbefore returning. But it calls through toget_pairwise_comparisons()with the defaulttest_type = "non_parametric", sowilcox.test()runs for every model pair (choose(n, 2)per metric per group) and the results are then thrown away.For callers that only want relative skill (which is what
add_relative_skill()is for), those tests are pure overhead. On the FluSight benchmark in #1221 that is ~16.4 s (2.4% of the eval-data build), all discarded. It also produces "cannot compute exact p-value with ties" warnings whenever the scores tie.Proposal: default
add_relative_skill(test_type = NULL), or otherwise let relative-skill-only callers skip the test. Sinceadd_relative_skill()discards the p-values regardless, this changes no output, only removing the wasted computation and the warnings.get_pairwise_comparisons()itself legitimately returns p-values, so its default is fine; this is specifically aboutadd_relative_skill()computing then discarding them.