Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions reflexio/server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -462,11 +462,12 @@ Key files:
- `shadow_comparison/judge.py`: Per-turn regular-vs-shadow judge
- `shadow_comparison/dispatcher.py` and `shadow_comparison/worker.py`: Publish-time dispatch and bounded background execution for shadow verdict writes
- `shadow_comparison/outcome.py`: Verdict outcome model helpers
- `evaluation_overview/service.py`: Aggregates evaluation-page metrics
- `evaluation_overview/service.py`: Bulk-loads evaluation-page metrics, first-request sources, citations, Braintrust scores, and optional shadow verdicts; source-set cohorts are computed server-side so the dashboard does not re-aggregate.
- `evaluation_overview/components/hero_state.py`, `evaluation_overview/components/distribution.py`, `evaluation_overview/components/rule_attribution.py`, `evaluation_overview/components/shadow_aggregation.py`: Focused aggregation helpers
- `evaluation_overview/eval_sampler.py`: Evaluation sampling helpers that remain root-level
- `models/api_schema/eval_overview_schema.py`: `GetEvaluationOverviewRequest/Response`, including optional `source_sets` input and `source_set_comparison` output keyed by `(user_id, session_id)`.

**Pattern**: Session-level agent success evaluation remains in `agent_success_evaluation/`; dashboard-facing rollups and per-turn shadow verdict analysis live in these companion directories.
**Pattern**: Session-level agent success evaluation remains in `agent_success_evaluation/`; dashboard-facing rollups, first-source cohort comparisons, and per-turn shadow verdict analysis live in these companion directories.

### Playbook Optimizer and Braintrust

Expand Down
8 changes: 6 additions & 2 deletions reflexio/server/services/evaluation_overview/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,18 @@

Read-side aggregation module for `POST /api/get_evaluation_overview`.

- `service.py` is the request-path entry point. It loads evaluation, citation, Braintrust, and optional shadow verdict data, then composes `GetEvaluationOverviewResponse`.
- `service.py` is the request-path entry point. It bulk-loads evaluation results (without embeddings), first-request sources, citations, Braintrust scores, and optional shadow verdicts, then composes `GetEvaluationOverviewResponse` for the dashboard.
- `components/` contains pure read-side aggregation helpers used by the service and focused tests.
- `eval_sampler.py` stays at the package root because regenerate jobs also use it to sample evaluation sessions.
- `reflexio/models/api_schema/eval_overview_schema.py` defines the public request/response contract, including optional `source_sets` cohorts and `source_set_comparison` output.

The overview reports task success across every evaluated session and a separate
behavior-success metric that excludes `failure_type=system_error` rows from
both numerator and denominator. A window with no behavior-evaluable rows
returns a null behavior rate plus eligible/excluded counts for honest UI
rendering.
rendering. Source-set comparison groups sessions by the first request source,
returns collision-safe `(user_id, session_id)` identities, and rejects duplicate
labels or overlapping source values so the frontend can compare cohorts without
client-side re-aggregation.

This module mutates no core state. Keep response-shape changes in API schema tests and service integration tests.
Loading