Skip to content

feat(frontend): migrate evaluator run invocation#3577

Open
mmabrouk wants to merge 9 commits intofeat/migrate-evaluator-playground-frontendfrom
feat/migrate-evaluator-playground-run
Open

feat(frontend): migrate evaluator run invocation#3577
mmabrouk wants to merge 9 commits intofeat/migrate-evaluator-playground-frontendfrom
feat/migrate-evaluator-playground-run

Conversation

@mmabrouk
Copy link
Member

@mmabrouk mmabrouk commented Jan 28, 2026

Summary

  • add workflow invoke helper for SimpleEvaluator execution
  • switch evaluator playground DebugSection to call /preview/workflows/invoke
  • update migration plan/status docs for PR2 run migration

Testing

  • pnpm lint-fix (warns about in web/oss/src/components/Editor/plugins/code/nodes/Base64Node.tsx)

Open with Devin

@vercel
Copy link

vercel bot commented Jan 28, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agenta-documentation Ready Ready Preview, Comment Feb 13, 2026 8:04pm

Request Review

@mmabrouk
Copy link
Member Author

Decisions & Tradeoffs

  • Use /preview/workflows/invoke from the evaluator playground instead of the legacy /evaluators/{key}/run. This matches the new backend path and keeps run logic aligned with SimpleEvaluator.
  • Fallback URI when evaluator not saved: if no config exists yet, build via . This keeps Run Evaluator working for unsaved drafts.
  • **Send parameters in both and **. The backend accepts both; this keeps compatibility with how workflow service requests are handled today.
  • Minimal response typing: only surface and since UI currently only needs evaluator outputs.

Alternatives considered

  • Keep legacy run endpoint (): lower change risk but blocks full migration; would keep old adapter path around.
  • Require saved evaluator config before run: strict but would break current UX; reduces ambiguity around URI.
  • **Only pass **: cleaner request but might miss expectations in existing workflow data handlers.
  • Expose trace_id/span_id and surface in UI: more debugging power but requires UI changes.

If you prefer a different tradeoff (e.g., require saved configs or drop ), I can adjust this PR.

Copy link
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 6 additional findings in Devin Review.

Open in Devin Review

@mmabrouk mmabrouk marked this pull request as ready for review February 13, 2026 19:55
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Feb 13, 2026
Expose output schemas from evaluator templates and send them on config create/edit, including dynamic derivation for auto_ai_critique and json_multi_field_match. Also remove legacy /evaluators/map usage and relax config listing filters so older non-human configs remain visible.
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Feb 13, 2026
@dosubot dosubot bot added the feature label Feb 13, 2026

evaluator_revision_slug = uuid4().hex[-12:]

hydrated_simple_evaluator_data = self._ensure_builtin_evaluator_data(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jp-agenta

We kept backend hydration on purpose, even after moving schema ownership to the frontend.
This PR makes the frontend send data.schemas.outputs when it can determine the schema at configure time. For fixed evaluators, the frontend reads outputs_schema from GET /evaluators. For auto_ai_critique, it derives the schema from parameters.json_schema.schema. For json_multi_field_match, it derives the schema from parameters.fields. If an evaluator has no known schema in the template, the frontend does not send one.
We still keep _ensure_builtin_evaluator_data in SimpleEvaluatorsService as a fallback. This protects non-UI callers and older payloads that still send only uri and parameters. It also keeps behavior compatible with the legacy flow, where builtin schemas were derived on the backend. In short, frontend schema sending is now the primary path, and backend hydration is a safety net.
This also means we did not remove existing behavior for old evaluators. We added a safer default while we complete the migration and verify all clients send schemas consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant