Hi dokimos-dev team — great to see a proper JUnit-native eval framework for the JVM side, and the multi-framework integration story (LangChain4j, Spring AI, Spring AI Alibaba, Koog, Embabel) is impressive.
I'm working on EvalPort, an open, portable schema for LLM evaluation results (validated by openeval.validate.validate_suite() / validate_result_set()). EvalPort's own adapters so far are all Python packages, and dokimos being Java/Kotlin means a native Java adapter isn't the natural fit here — but I don't think that has to block interop.
Reading dokimos-core/src/main/java/dev/dokimos/core/EvalResult.java and ExperimentResult.java, you already have exactly the kind of well-defined native result model this is built around:
EvalResult (record): name, score, threshold, success, reason, metadataExperimentResult (record): runResults: List<RunResult>, plus passRate(), averageScore(evaluatorName), scoreStdDev(evaluatorName), evaluatorNames(), and — notably — exportJson(Path) / toJson() already built in.
Since ExperimentResult.toJson()/exportJson() already gives a stable, documented JSON export, my proposal is: a separate Python adapter package (evalport-dokimos or similar) that reads that exported JSON and converts it to/from the EvalPort schema, rather than us writing any Java/Kotlin code into this repo. That keeps dokimos itself untouched — the adapter would just need the JSON export's shape documented (or a small fixture/example file) to build against.
Happy to build that adapter myself if the exported JSON schema is stable/documented somewhere, or to collaborate if you'd rather define the mapping together. Totally low-pressure either way — just flagging the idea in case it's useful for teams that want to compare JVM-side eval runs against results from Python-side tools.
Spec: https://github.com/adhabnr-ux/evalport
Hi dokimos-dev team — great to see a proper JUnit-native eval framework for the JVM side, and the multi-framework integration story (LangChain4j, Spring AI, Spring AI Alibaba, Koog, Embabel) is impressive.
I'm working on EvalPort, an open, portable schema for LLM evaluation results (validated by
openeval.validate.validate_suite()/validate_result_set()). EvalPort's own adapters so far are all Python packages, and dokimos being Java/Kotlin means a native Java adapter isn't the natural fit here — but I don't think that has to block interop.Reading
dokimos-core/src/main/java/dev/dokimos/core/EvalResult.javaandExperimentResult.java, you already have exactly the kind of well-defined native result model this is built around:EvalResult(record):name,score,threshold,success,reason,metadataExperimentResult(record):runResults: List<RunResult>, pluspassRate(),averageScore(evaluatorName),scoreStdDev(evaluatorName),evaluatorNames(), and — notably —exportJson(Path)/toJson()already built in.Since
ExperimentResult.toJson()/exportJson()already gives a stable, documented JSON export, my proposal is: a separate Python adapter package (evalport-dokimosor similar) that reads that exported JSON and converts it to/from the EvalPort schema, rather than us writing any Java/Kotlin code into this repo. That keeps dokimos itself untouched — the adapter would just need the JSON export's shape documented (or a small fixture/example file) to build against.Happy to build that adapter myself if the exported JSON schema is stable/documented somewhere, or to collaborate if you'd rather define the mapping together. Totally low-pressure either way — just flagging the idea in case it's useful for teams that want to compare JVM-side eval runs against results from Python-side tools.
Spec: https://github.com/adhabnr-ux/evalport