feat: derive corrective turns from session judge - #401
Conversation
Count corrective user turns in the agent-success structured output, persist the judge value across evaluation flows, and preserve regenerated results for both insert- and upsert-style storage backends.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 29 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAgent-success evaluation now uses prompt v1.1.0 to produce validated corrective-turn counts, propagates those counts from LLM output, and reconciles regenerated evaluation rows for both insert and in-place upsert storage. ChangesAgent success evaluation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant PromptV110
participant AgentSuccessEvaluator
participant EvaluationStorage
PromptV110->>AgentSuccessEvaluator: returns corrective-turn count
AgentSuccessEvaluator->>EvaluationStorage: saves evaluation result
AgentSuccessEvaluator->>EvaluationStorage: reconciles prior and post-save result IDs
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
reflexio/server/services/agent_success_evaluation/runner.py (1)
314-318: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueInline step numbering drifted from the docstring.
The docstring (Lines 115-119) labels the delete/reconcile logic as step 7 (step 6 = "Run evaluation service"), but this comment still says
# 6..✏️ Fix
- # 6. New rows saved successfully. Delete captured prior rows only when the + # 7. New rows saved successfully. Delete captured prior rows only when the🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@reflexio/server/services/agent_success_evaluation/runner.py` around lines 314 - 318, Correct the inline step number above the old-result deletion logic guarded by old_result_ids from step 6 to step 7, matching the surrounding function docstring. Leave the deletion and reconciliation behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@reflexio/server/prompt/prompt_bank/agent_success_evaluation/v1.1.0.prompt.md`:
- Around line 90-93: Update the failure example in the agent success evaluation
prompt so the failure_type field contains one concrete valid enum value, such as
"missing_tool", instead of an inline “or” expression; describe the other allowed
values in prose while preserving the remaining JSON structure.
In `@reflexio/server/services/agent_success_evaluation/runner.py`:
- Around line 254-268: Update the force_regenerate setup in the evaluation
runner to derive evaluation_name from root_config.agent_success_config, using
the nested AgentSuccessConfig singleton expected by the evaluator. Keep the
existing prior-result ID lookup unchanged so regeneration cleanup targets the
same rows written by the agent-success evaluator.
---
Nitpick comments:
In `@reflexio/server/services/agent_success_evaluation/runner.py`:
- Around line 314-318: Correct the inline step number above the old-result
deletion logic guarded by old_result_ids from step 6 to step 7, matching the
surrounding function docstring. Leave the deletion and reconciliation behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e83cc40f-8937-480f-b393-ee710b6ebb8a
📒 Files selected for processing (15)
reflexio/models/api_schema/domain/entities.pyreflexio/models/api_schema/eval_overview_schema.pyreflexio/models/api_schema/ui/entities.pyreflexio/server/prompt/prompt_bank/agent_success_evaluation/v1.0.0.prompt.mdreflexio/server/prompt/prompt_bank/agent_success_evaluation/v1.1.0.prompt.mdreflexio/server/services/agent_success_evaluation/agent_success_evaluation_constants.pyreflexio/server/services/agent_success_evaluation/components/evaluator.pyreflexio/server/services/agent_success_evaluation/runner.pyreflexio/test_support/llm_model_registry.pytests/fixtures/llm/agent_success_evaluation.jsontests/server/services/__snapshots__/test_llm_mock_schema_compliance/TestMockResponseSnapshots.test_recorded_fixture_content[agent_success_evaluation].jsontests/server/services/agent_success_evaluation/test_agent_success_evaluation_utils.pytests/server/services/agent_success_evaluation/test_agent_success_evaluator.pytests/server/services/agent_success_evaluation/test_group_evaluation_runner_regen.pytests/server/services/test_prompt_model_mapping.py
Use the agent-success singleton config during regeneration cleanup and keep prompt output examples valid JSON. Add focused regressions for both contracts.
|
CodeRabbit follow-up: fixed both inline findings and the review-body step-numbering nitpick in cd549b5. The focused suite passes (89 passed, 2 skipped), with clean Ruff, formatting, and Pyright checks. |
1 participant
Summary
Changes
Evaluation contract
number_of_correction_per_sessionin the structured judge response.AgentSuccessEvaluationResultand remove playbook-count lookup/fallback behavior.Persistence and regeneration
Schemas and tests
Test Plan
uv run ruff checkon all changed Python filesuv run pyrighton all changed Python filesuv run pytest -q --no-cov tests/server/services/agent_success_evaluation/test_agent_success_evaluator.py tests/server/services/agent_success_evaluation/test_agent_success_evaluation_utils.py tests/server/services/agent_success_evaluation/test_group_evaluation_runner_regen.py tests/server/services/test_prompt_model_mapping.pyFollow-ups
cd549b5a: valid JSON output examples, nested agent-success config lookup during regeneration, and corrected runner step numbering.Summary by CodeRabbit
New Features
Bug Fixes