feat(compare): add confidence intervals to comparison output #41
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes permission check bypass where `exit 0` only stopped the check step but allowed subsequent steps to continue running. Unauthorized users could trigger Claude Code reviews by opening PRs. Changes: - Add step ID and output flags (authorized=true/false) - Gate all subsequent steps with `if:` condition on authorization Closes #39 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add optional confidenceIntervals fields to comparison metrics schemas, computed via bootstrap sampling when strategy=statistical. This exposes uncertainty bounds for aggregate metrics to help assess statistical significance. Changes: - Add ConfidenceIntervalSchema and extend QualityMetrics, PerformanceMetrics, TrialsCapabilityMetrics, and TrialsReliabilityMetrics schemas - Create shared bootstrap utility (src/graders/bootstrap.ts) with configurable iterations and confidence level - Refactor compare-statistical.ts and trials-compare-statistical.ts to use shared bootstrap module - Add CI computation in compare.ts and compare-trials.ts for statistical strategy - Update markdown formatters to display 95% CI columns when present - Add comprehensive unit tests for bootstrap utility Closes #39 Closes #40 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename `mean` to `median` in BootstrapResult for semantic clarity (the value is the 50th percentile of bootstrap means, not arithmetic mean) - Extract duplicate `formatCI` function to shared bootstrap.ts module - Remove orphaned TSDoc comment in compare.ts - Add comprehensive integration tests for statistical strategy CI computation - Move bootstrap.spec.ts to src/graders/tests/ for consistent organization - Fix package.json script ordering (formatter cleanup) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Document the new confidence intervals feature from PR #41: - Add CI output examples to SKILL.md for both CaptureResult and TrialResult - Update comparison-graders.md with detailed statistical strategy output - Document markdown output format with 95% CI columns Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
confidenceIntervalsfields to comparison metric schemasstrategy=statisticalChanges
src/graders/bootstrap.ts— shared bootstrap utility with configurable iterations/confidence levelsrc/graders/bootstrap.spec.ts— 17 unit tests for bootstrap utilityConfidenceIntervalSchemaand CI fields inQualityMetrics,PerformanceMetrics,TrialsCapabilityMetrics,TrialsReliabilityMetricscompare.tsandcompare-trials.tsfor statistical strategyUsage
Output includes:
Test plan
bun run checkpasses (type/lint/format)bun testpasses (489 tests)--strategy statisticalCloses #39
Closes #40
🤖 Generated with Claude Code