don't error if baseline model is not present

[sbfnk]

avoids the pairwise comparison erroring if the specified baseline model is not present in the data (happening in some instance at the European Forecast Hub).

It could also make sense to throw an informative error. The function can still compute useable pairwise comparisons so I thought a warning might be more appropriate here.

[codecov]

Codecov Report

Merging #250 (c54d371) into master (1bffc42) will decrease coverage by 0.12%.
The diff coverage is 33.33%.

❗ Current head c54d371 differs from pull request most recent head 527b5b4. Consider uploading reports for the commit 527b5b4 to get more accurate results

@@            Coverage Diff             @@
##           master     #250      +/-   ##
==========================================
- Coverage   91.32%   91.20%   -0.13%     
==========================================
  Files          22       22              
  Lines        1361     1364       +3     
==========================================
+ Hits         1243     1244       +1     
- Misses        118      120       +2     

Impacted Files	Coverage Δ
R/pairwise-comparisons.R	85.82% <33.33%> (-1.28%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sbfnk]

On reflection and now that the immediate problem has been fixed I think it makes more sense to error here as baseline will have been set to not-NULL. I've force pushed an update.

[nikosbosse]

Could you please clarify again what should happen with this PR?

[sbfnk]

Could you please clarify again what should happen with this PR?

As far as I'm concerned it's ready to be merged.

[nikosbosse]

The test was failing, so I updated it. Just to confirm: before the function was failing with a cryptic message and now there is an explicit error message, right?
It might be better to have a check at an earlier point (because this is checking whether the baseline model is there right where the computation happens, and might return "baseline missing" even if the error is actually something else. Don't know, maybe also not a big deal

[sbfnk]

It might be better to have a check at an earlier point (because this is checking whether the baseline model is there right where the computation happens, and might return "baseline missing" even if the error is actually something else. Don't know, maybe also not a big deal

There is already a check at an earlier point

scoringutils/R/summarise_scores.R

Line 187 in 3e1a9d6

if (!is.null(baseline) && !(baseline %in% models)) {

however this is only done when scores are created using summarise_scores. If pairwise_comparison is called directly as e.g. in https://github.com/covid19-forecast-hub-europe/EuroForecastHub/blob/e2dc0c97f379f2001e57ad84853df7577df59d9a/R/summarise_scores.R#L79 then this is not done.

What I'm not clear on is whether this could have been implemented using summarise_scores, in which case it might make sense not to export pairwise_comparison, ensuring checks are always done (and the one added in this PR could be removed again).