There should be a warning when there is a column called "sample" with a quantile format

[nikosbosse]

I constructed some synthetic data and run into an error that was tricky to spot, caused by the fact that the sample column was automatically ignored:

library(dplyr)
library(tidyr)
library(scoringutils)


n_sim <- 1000
epsilon <- rnorm(n_sim)
Y <- exp(epsilon)

forecasts <- expand.grid(
  sigma = 1:20/10, 
  quantile = c(0.01, 0.025, 1:19/20, 0.975, 0.99)
)

forecasts <- forecasts |>
  as_tibble() |>
  mutate(model = 10 * sigma, 
         prediction = exp(qnorm(quantile, sd = sigma)), 
         true_value = list(Y), 
         sample = list(1:length(Y))) |>
  unnest(c(true_value, sample))

check_forecasts(forecasts)
score(forecasts)

[seabbs]

It seems like it might be better to improve the handling of forecast unit detection vs adding documentation. That means making get_forecast_unit a bit smarter.

Full reprex:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(scoringutils)
#> Note: scoringutils 1.0.0 introduces a lot of breaking changes and we apologise for any inconvenience. If you prefer the old interface, please download version 0.1.8 using remotes::install_github("epiforecasts/scoringutils@v0.1.8")


n_sim <- 1000
epsilon <- rnorm(n_sim)
Y <- exp(epsilon)

forecasts <- expand.grid(
  sigma = 1:20/10, 
  quantile = c(0.01, 0.025, 1:19/20, 0.975, 0.99)
)

forecasts <- forecasts |>
  as_tibble() |>
  mutate(model = 10 * sigma,
         prediction = exp(qnorm(quantile, sd = sigma)),
         true_value = list(Y),
         sample = list(1:length(Y))) |>
  unnest(c(true_value, sample))

check_forecasts(forecasts)
#> Your forecasts seem to be for a target of the following type:
#> $target_type
#> [1] "continuous"
#> 
#> and in the following format:
#> $prediction_type
#> [1] "quantile"
#> 
#> The unit of a single forecast is defined by:
#> $forecast_unit
#> [1] "sigma" "model"
#> 
#> Cleaned data, rows with NA values in prediction or true_value removed:
#> $cleaned_data
#>         sigma quantile model  prediction true_value sample
#>         <num>    <num> <num>       <num>      <num>  <int>
#>      1:   0.1     0.01     1   0.7924429  0.7691318      1
#>      2:   0.1     0.01     1   0.7924429  0.2793781      2
#>      3:   0.1     0.01     1   0.7924429  1.2427315      3
#>      4:   0.1     0.01     1   0.7924429  1.6908452      4
#>      5:   0.1     0.01     1   0.7924429  1.1035929      5
#>     ---                                                   
#> 459996:   2.0     0.99    20 104.8673007  0.2715035    996
#> 459997:   2.0     0.99    20 104.8673007  1.3315106    997
#> 459998:   2.0     0.99    20 104.8673007  0.5470439    998
#> 459999:   2.0     0.99    20 104.8673007  0.1263871    999
#> 460000:   2.0     0.99    20 104.8673007  2.4062132   1000
#> 
#> Number of unique values per column per model:
#> $unique_values
#>     model sigma quantile prediction true_value sample
#>     <num> <int>    <int>      <int>      <int>  <int>
#>  1:     1     1       23         23       1000   1000
#>  2:     2     1       23         23       1000   1000
#>  3:     3     1       23         23       1000   1000
#>  4:     4     1       23         23       1000   1000
#>  5:     5     1       23         23       1000   1000
#>  6:     6     1       23         23       1000   1000
#>  7:     7     1       23         23       1000   1000
#>  8:     8     1       23         23       1000   1000
#>  9:     9     1       23         23       1000   1000
#> 10:    10     1       23         23       1000   1000
#> 11:    11     1       23         23       1000   1000
#> 12:    12     1       23         23       1000   1000
#> 13:    13     1       23         23       1000   1000
#> 14:    14     1       23         23       1000   1000
#> 15:    15     1       23         23       1000   1000
#> 16:    16     1       23         23       1000   1000
#> 17:    17     1       23         23       1000   1000
#> 18:    18     1       23         23       1000   1000
#> 19:    19     1       23         23       1000   1000
#> 20:    20     1       23         23       1000   1000
#>     model sigma quantile prediction true_value sample
score(forecasts)
#> Error in if (true_value == median_prediction) {: the condition has length > 1

^{Created on 2023-01-12 with reprex v2.0.2}