Skip to content

Comments

Add more statistics to the output table#28

Merged
roninsightrx merged 3 commits intoincremental-bayesfrom
ofv-r2
Oct 10, 2025
Merged

Add more statistics to the output table#28
roninsightrx merged 3 commits intoincremental-bayesfrom
ofv-r2

Conversation

@roninsightrx
Copy link
Contributor

@roninsightrx roninsightrx commented Oct 9, 2025

Implements #21 . The original request was r^2, but sum-of-squares makes more sense.
OFV and weighted sum of squares we need for model averaging (#27 ), on their own they are not so useful as a statistic.

Either OFV or SS can now easily be used to compute a weighted prediction (=model averaging).

@roninsightrx roninsightrx changed the base branch from main to incremental-bayes October 9, 2025 23:48
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds additional statistics (OFV and weighted sum-of-squares) to the output table to support model averaging and provide more comprehensive evaluation metrics.

  • Introduces a new ss() function to calculate weighted sum-of-squares of residuals
  • Adds ofv and ss_w columns to the results output table
  • Updates tests to verify the new columns and their calculated values

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
R/misc.R Implements the new ss() function for weighted sum-of-squares calculation
R/run_eval_core.R Integrates OFV and weighted sum-of-squares into the results table
man/ss.Rd Documentation for the new ss() function
tests/testthat/test-run_eval.R Test updates to verify new statistics columns and values

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Copy link
Collaborator

@mccarthy-m-g mccarthy-m-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just some minor suggestions:

Comment on lines +71 to +85
#' Weighted sum-of-squares of residuals
#'
#' @inheritParams rmse
#' @param w weights
#'
ss <- function(obs, pred, w = NULL) {
if(is.null(w)) {
w <- rep(1, length(obs))
}
if (length(obs) != length(pred) || length(obs) != length(w)) {
cli::cli_abort("`obs`, `pred`, and `w` must have the same length")
}
if(sum(w) == 0) return(NA)
sum(w * (obs - pred)^2)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion for something more concise (you can take or leave). vctrs::vec_recycle_common() will throw an informative error if any of the lengths are different, e.g.:

ss(obs = 1:5, pred = 5:1, w = 1:3)
#> Error in `ss()`:
#> ! Can't recycle `obs` (size 5) to match `w` (size 3).
Suggested change
#' Weighted sum-of-squares of residuals
#'
#' @inheritParams rmse
#' @param w weights
#'
ss <- function(obs, pred, w = NULL) {
if(is.null(w)) {
w <- rep(1, length(obs))
}
if (length(obs) != length(pred) || length(obs) != length(w)) {
cli::cli_abort("`obs`, `pred`, and `w` must have the same length")
}
if(sum(w) == 0) return(NA)
sum(w * (obs - pred)^2)
}
#' Weighted sum-of-squares of residuals
#'
#' @inheritParams rmse
#' @param w A numeric vector of weights the same size as `obs` and `pred`.
#' Vectors of size 1 will be recycled to the size of `obs` and `pred`.
#'
ss <- function(obs, pred, w = 1) {
args <- vctrs::vec_recycle_common(obs = obs, pred = pred, w = w)
if(sum(args[["w"]]) == 0) return(NA)
sum(args[["w"]] * (args[["obs"]] - args[["pred"]])^2)
}

Here are some tests you could add for this too:

test_that("recycling + incompatible size error works", {
  expect_equal(ss(obs = 1:3, pred = 3:1, w = 1), 8)
  expect_equal(ss(obs = 1:3, pred = 3:1, w = c(1, 1, 0.5)), 6)
  expect_error(
    ss(obs = 1:5, pred = 5:1, w = 1:3), class = "vctrs_error_incompatible_size"
  )
})

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>
@roninsightrx
Copy link
Contributor Author

Looks good! Just some minor suggestions:

in principle good suggestion, but leaving as-is for now for consistency (we should then also update the other stats functions, rmse etc to use the same approach)

@roninsightrx roninsightrx merged commit 4f53158 into incremental-bayes Oct 10, 2025
@roninsightrx roninsightrx deleted the ofv-r2 branch October 10, 2025 19:13
roninsightrx added a commit that referenced this pull request Oct 21, 2025
* add more stats

* add tests + update docs

* fix tests

* minor build fixes

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* Apply suggestion from @mccarthy-m-g

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

* move to S3 + print function + few more stats

* fix tests

* add incremental option

* split off weighting function

* fix docs + print function

* add more testing

* add tests for handle_sample_weighting()

* Add more statistics to the output table (#28)

* add ofv and ss

* add test

* Update R/run_eval_core.R

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

---------

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>

---------

Co-authored-by: Michael McCarthy <51542091+mccarthy-m-g@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[LOW] export the OFV and sum of squares from the MAP fit in the raw results

2 participants