feat(frontend): migrate evaluator run invocation#3577
feat(frontend): migrate evaluator run invocation#3577mmabrouk wants to merge 9 commits intofeat/migrate-evaluator-playground-frontendfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Decisions & Tradeoffs
Alternatives considered
If you prefer a different tradeoff (e.g., require saved configs or drop ), I can adjust this PR. |
...ponents/pages/evaluations/autoEvaluation/EvaluatorsModal/ConfigureEvaluator/DebugSection.tsx
Show resolved
Hide resolved
…igrate-evaluator-playground-run
This reverts commit 00c2aa2.
Expose output schemas from evaluator templates and send them on config create/edit, including dynamic derivation for auto_ai_critique and json_multi_field_match. Also remove legacy /evaluators/map usage and relax config listing filters so older non-human configs remain visible.
|
|
||
| evaluator_revision_slug = uuid4().hex[-12:] | ||
|
|
||
| hydrated_simple_evaluator_data = self._ensure_builtin_evaluator_data( |
There was a problem hiding this comment.
We kept backend hydration on purpose, even after moving schema ownership to the frontend.
This PR makes the frontend send data.schemas.outputs when it can determine the schema at configure time. For fixed evaluators, the frontend reads outputs_schema from GET /evaluators. For auto_ai_critique, it derives the schema from parameters.json_schema.schema. For json_multi_field_match, it derives the schema from parameters.fields. If an evaluator has no known schema in the template, the frontend does not send one.
We still keep _ensure_builtin_evaluator_data in SimpleEvaluatorsService as a fallback. This protects non-UI callers and older payloads that still send only uri and parameters. It also keeps behavior compatible with the legacy flow, where builtin schemas were derived on the backend. In short, frontend schema sending is now the primary path, and backend hydration is a safety net.
This also means we did not remove existing behavior for old evaluators. We added a safer default while we complete the migration and verify all clients send schemas consistently.
Summary
/preview/workflows/invokeTesting