[Proposal] EvalPort Import/Export Support

[adhabnr-ux]

Overview

We have launched EvalPort (v1.0.0-rc.1), an open standard (Apache 2.0) for portable LLM evaluation datasets. Every eval framework uses incompatible formats — EvalPort provides a standard JSON format + converters + SDKs (TypeScript + Python).

Note: Formerly called "OpenEval" — renamed to avoid confusion with open-eval.com (a Johns Hopkins/Microsoft Research project). The PR title and dependency name need updating accordingly.

What We're Proposing

Add to_evalport() and from_evalport() methods (~50-100 lines using our SDK).

Resources

• Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
• npm SDK: npm install evalport-sdk
• PyPI SDK: pip install evalport-sdk

We would love to collaborate.

[DresdenGman]

I can take this. I will keep it narrow and start by checking the OpenEval spec and the existing eval data model in Autogen, then propose the smallest to_openeval() / from_openeval() surface that fits the current package boundaries. If there is a preferred package or serializer location, please point me there and I will align with it before opening a PR.

[adhabnr-ux]

Update: OpenEval packages are now published!

• npm: npm install openeval-sdk and npm install -g openeval-cli
• PyPI: pip install openeval-sdk
• GitHub: https://github.com/adhabnr-ux/openeval

We have active responses from Inspect AI, CrewAI, and Arize/OpenInference contributors. Would love to collaborate on native import/export support here too.

[adhabnr-ux]

Awesome! 🎉

Here's what I'd suggest for the package/serializer location:

For to_openeval() / from_openeval() in AutoGen, the cleanest approach would be:

1. Create a new module autogen/eval/openeval.py (or add to an existing eval module if one exists)
2. Use our Python SDK as a dependency: pip install openeval-sdk
3. Minimal surface — just two functions:

from openeval.convert import create_result_set
from openeval.types import OPENEVAL_VERSION

def to_openeval(autogen_eval_result) -> dict:
    """Export AutoGen eval results to OpenEval ResultSet format."""
    test_cases = []
    for result in autogen_eval_result.results:
        tc = {
            "id": result.task_id,
            "input": result.task_description,
            "expected_output": result.expected_output,
            "graders": ["gr_output_match"],
            "expected_tools": result.expected_tools or [],
        }
        test_cases.append(tc)
    return {
        "version": OPENEVAL_VERSION,
        "id": f"autogen_eval_{autogen_eval_result.run_id}",
        "test_cases": test_cases,
        "graders": [{"id": "gr_output_match", "type": "exact_match", "params": {"ignore_case": True}}],
    }

def from_openeval(suite: dict) -> list:
    """Import OpenEval suite into AutoGen's eval format."""
    return [
        {"task_id": tc["id"], "description": tc["input"], "expected_output": tc.get("expected_output", "")}
        for tc in suite.get("test_cases", [])
    ]

4. AutoGen's agent eval maps well to OpenEval — expected_tools field for tool-call verification, llm_judge graders for response quality.

5. Tests — add a test that converts a simple AutoGen eval to OpenEval and validates it with from openeval.validate import validate_suite.

Let me know if you need any changes to the spec or SDK to make this fit better. I'm happy to review your PR!

Resources:

• Spec: https://github.com/adhabnr-ux/openeval/blob/main/spec/SPEC.md
• Python SDK: pip install openeval-sdk
• Agent eval example: https://github.com/adhabnr-ux/openeval/blob/main/examples/agent-tools.json

[DresdenGman]

I have a draft PR open now: #8009

It follows the suggested module boundary in autogenstudio/eval, keeps the surface to to_openeval() / from_openeval(), and includes a small round-trip regression test. I also added openeval-sdk as the runtime dependency for autogenstudio.

If you want a different mapping for expected_tools or a different package location, I can adjust before merging.

[adhabnr-ux]

Heads up Dresden (@DresdenGman) — we've renamed from OpenEval to EvalPort to avoid a name collision with open-eval.com (a Johns Hopkins/Microsoft Research project). The Python package is now evalport-sdk on PyPI (https://pypi.org/project/evalport-sdk/1.0.0/).

You'll need to update the dependency in your PR from openeval-sdk to evalport-sdk. The Python import stays from openeval.validate import... for backward compatibility — only the package name changed.

GitHub repo: https://github.com/adhabnr-ux/evalport

Sorry for the churn — better to do this now before merge than after!

[DresdenGman]

Thanks for the heads-up. I updated PR #8009 to use the renamed evalport-sdk distribution and regenerated python/uv.lock. The Python imports remain under openeval as requested. I also verified that evalport-sdk==1.0.0 provides both openeval.types and openeval.validate.