Add more statistics to the output table#28
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds additional statistics (OFV and weighted sum-of-squares) to the output table to support model averaging and provide more comprehensive evaluation metrics.
- Introduces a new
ss()function to calculate weighted sum-of-squares of residuals - Adds
ofvandss_wcolumns to the results output table - Updates tests to verify the new columns and their calculated values
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| R/misc.R | Implements the new ss() function for weighted sum-of-squares calculation |
| R/run_eval_core.R | Integrates OFV and weighted sum-of-squares into the results table |
| man/ss.Rd | Documentation for the new ss() function |
| tests/testthat/test-run_eval.R | Test updates to verify new statistics columns and values |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
mccarthy-m-g
left a comment
There was a problem hiding this comment.
Looks good! Just some minor suggestions:
| #' Weighted sum-of-squares of residuals | ||
| #' | ||
| #' @inheritParams rmse | ||
| #' @param w weights | ||
| #' | ||
| ss <- function(obs, pred, w = NULL) { | ||
| if(is.null(w)) { | ||
| w <- rep(1, length(obs)) | ||
| } | ||
| if (length(obs) != length(pred) || length(obs) != length(w)) { | ||
| cli::cli_abort("`obs`, `pred`, and `w` must have the same length") | ||
| } | ||
| if(sum(w) == 0) return(NA) | ||
| sum(w * (obs - pred)^2) | ||
| } |
There was a problem hiding this comment.
Suggestion for something more concise (you can take or leave). vctrs::vec_recycle_common() will throw an informative error if any of the lengths are different, e.g.:
ss(obs = 1:5, pred = 5:1, w = 1:3)
#> Error in `ss()`:
#> ! Can't recycle `obs` (size 5) to match `w` (size 3).| #' Weighted sum-of-squares of residuals | |
| #' | |
| #' @inheritParams rmse | |
| #' @param w weights | |
| #' | |
| ss <- function(obs, pred, w = NULL) { | |
| if(is.null(w)) { | |
| w <- rep(1, length(obs)) | |
| } | |
| if (length(obs) != length(pred) || length(obs) != length(w)) { | |
| cli::cli_abort("`obs`, `pred`, and `w` must have the same length") | |
| } | |
| if(sum(w) == 0) return(NA) | |
| sum(w * (obs - pred)^2) | |
| } | |
| #' Weighted sum-of-squares of residuals | |
| #' | |
| #' @inheritParams rmse | |
| #' @param w A numeric vector of weights the same size as `obs` and `pred`. | |
| #' Vectors of size 1 will be recycled to the size of `obs` and `pred`. | |
| #' | |
| ss <- function(obs, pred, w = 1) { | |
| args <- vctrs::vec_recycle_common(obs = obs, pred = pred, w = w) | |
| if(sum(args[["w"]]) == 0) return(NA) | |
| sum(args[["w"]] * (args[["obs"]] - args[["pred"]])^2) | |
| } |
Here are some tests you could add for this too:
test_that("recycling + incompatible size error works", {
expect_equal(ss(obs = 1:3, pred = 3:1, w = 1), 8)
expect_equal(ss(obs = 1:3, pred = 3:1, w = c(1, 1, 0.5)), 6)
expect_error(
ss(obs = 1:5, pred = 5:1, w = 1:3), class = "vctrs_error_incompatible_size"
)
})Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>
in principle good suggestion, but leaving as-is for now for consistency (we should then also update the other stats functions, |
* add more stats * add tests + update docs * fix tests * minor build fixes * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * Apply suggestion from @mccarthy-m-g Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> * move to S3 + print function + few more stats * fix tests * add incremental option * split off weighting function * fix docs + print function * add more testing * add tests for handle_sample_weighting() * Add more statistics to the output table (#28) * add ofv and ss * add test * Update R/run_eval_core.R Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> --------- Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com> --------- Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>
Implements #21 . The original request was r^2, but sum-of-squares makes more sense.
OFV and weighted sum of squares we need for model averaging (#27 ), on their own they are not so useful as a statistic.
Either OFV or SS can now easily be used to compute a weighted prediction (=model averaging).