Proposal: EvalPort adapter for testing/_judge.py's Judge/LLMJudge (open interchange format for eval results) #638

Summary

I maintain EvalPort, an Apache-2.0, framework-agnostic interchange format for LLM evaluation test cases, graders, and results (Python/TS SDK evalport-sdk, 37 adapter packages for tools like DeepEval, Braintrust, Ragas, LangSmith, MLflow, etc.). I'd like to propose a small standalone adapter package — vision-agents-openeval-adapter — that converts the output of this repo's Judge/LLMJudge (in agents-core/vision_agents/testing/_judge.py) into EvalPort's GraderResult/ResultSet shape, so intent-eval runs recorded here can be diffed, archived, or compared against grader output from other frameworks using a common format.

This is a proposal, not a PR — I want to check whether this is something you'd want at all, and if so, whether an out-of-tree package (like the pattern below) or something else entirely fits this project's direction better, before writing any code against your API.

Why I think this maps well

I read _judge.py directly (not docs) to confirm the real shapes:

@dataclass
class JudgeVerdict:
    success: bool
    reason: str

class Judge(Protocol):
    async def evaluate(self, event: ChatMessageEvent, intent: str) -> JudgeVerdict: ...

class LLMJudge:
    def __init__(self, llm: LLM) -> None: ...
    async def evaluate(self, event: ChatMessageEvent, intent: str) -> JudgeVerdict: ...

JudgeVerdict is structurally close to EvalPort's GraderResult (grader_id, score, passed, reason, metadata): success → passed, reason → reason directly. The one real gap is that JudgeVerdict is binary pass/fail with no numeric score — EvalPort's schema allows score to be null, so a straightforward mapping would leave score: null and set passed from success rather than inventing a number that isn't there. intent maps naturally to an EvalPort TestCase.expected_output (or a dedicated intent field carried in metadata), and event.content to the case's actual_output.

A rough sketch of what the adapter-side function would look like, using only the real fields above:

# vision_agents_openeval_adapter/__init__.py
from vision_agents.testing._judge import JudgeVerdict
from vision_agents.testing import ChatMessageEvent

def verdict_to_grader_result(
    verdict: JudgeVerdict,
    *,
    grader_id: str = "vision_agents_llm_judge",
    event: ChatMessageEvent | None = None,
    intent: str | None = None,
) -> dict:
    """Convert a vision_agents JudgeVerdict into an EvalPort GraderResult."""
    return {
        "grader_id": grader_id,
        "score": None,  # LLMJudge is pass/fail only, no numeric confidence
        "passed": verdict.success,
        "reason": verdict.reason,
        "metadata": {
            "vision_agents": {
                "intent": intent,
                "message": getattr(event, "content", None),
            }
        },
    }

Comparable adapters already shipped

Two adapters in EvalPort follow close variants of this shape today, which is what makes me think this is a reasonable fit rather than a stretch:

• deepeval-openeval-adapter converts DeepEval's MetricData (name, score, success, reason) into GraderResult almost field-for-field — the same success → passed, reason → reason mapping I'd use here.
• braintrust-openeval-adapter is the template for "standalone package, zero footprint on the target repo" — it works entirely from Braintrust's public result shape from the outside, nothing merged into braintrust itself. I'd follow the same pattern here rather than presuming this repo wants an in-tree module: the adapter would depend on vision-agents (for the real JudgeVerdict/ChatMessageEvent types), not the other way around.

Questions before I'd write real code

1. Is this useful to you at all, given _judge.py is currently intended for this repo's own testing guide rather than as a public grading API? I don't want to propose scope creep on an internal testing utility.
2. If it is useful — standalone adapter package (my default plan, no footprint here), or would you rather this live as an optional extra inside vision-agents itself?
3. Is JudgeVerdict considered stable enough to build against, or is it likely to gain fields (e.g. a numeric confidence score) soon, which would change the mapping above?

Happy to build and test this against the real vision-agents package (mirroring how the DeepEval/Braintrust adapters test against the real installed SDKs, not mocks) if there's interest. No obligation either way — thanks for reading this far, and for open-sourcing the testing harness in the first place.

— Sahi, independent contributor (not affiliated with GetStream)