Skip to content

Comments

fix: RubricGroup scoring_ms should be the time to score all rubrics#953

Open
pjo256 wants to merge 2 commits intoPrimeIntellect-ai:mainfrom
pjo256:fix/rubric-group-timing
Open

fix: RubricGroup scoring_ms should be the time to score all rubrics#953
pjo256 wants to merge 2 commits intoPrimeIntellect-ai:mainfrom
pjo256:fix/rubric-group-timing

Conversation

@pjo256
Copy link

@pjo256 pjo256 commented Feb 22, 2026

Description

Measure total scoring time once at the RubricGroup level. scoring_ms was previously the last rubric's scoring_ms in a RubricGroup.

Fixes #952.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Test improvement

Testing

  • All existing tests pass when running uv run pytest locally.
  • New tests have been added to cover the changes

Checklist

  • My code follows the style guidelines of this project as outlined in AGENTS.md
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Additional Notes


Note

Low Risk
Low risk: isolated change to timing bookkeeping in RubricGroup, plus tests validating scoring_ms/total_ms consistency.

Overview
Fixes RubricGroup timing so scoring_ms reflects the total time spent scoring across all rubrics (rather than whatever the last rubric wrote). score_rollout and score_group now measure wall-clock time once at the group level, restore each state’s original timing between rubric evaluations, and then update scoring_ms and increment total_ms after aggregation.

Adds async tests asserting generation_ms + scoring_ms == total_ms for both score_rollout and score_group.

Written by Cursor Bugbot for commit 67c3e20. This will update automatically on new commits. Configure here.

Signed-off-by: Philip Ottesen <phiott256@gmail.com>
Signed-off-by: Philip Ottesen <phiott256@gmail.com>
@pjo256 pjo256 changed the title fix: scoring_ms should be the total scoring time for all rubrics fix: RubricGroup scoring_ms should be the total scoring time for all rubrics Feb 22, 2026
@pjo256 pjo256 changed the title fix: RubricGroup scoring_ms should be the total scoring time for all rubrics fix: RubricGroup scoring_ms should be the time to score all rubrics Feb 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inaccurate rubric timings when using RubricGroup

1 participant