Types as first class citizens

[stefanks-px]

Hi,
first of all thanks for bringing Dokimos to life. I really like the lib and we started using it for offline evaluations in our company.

One thing that I'm curious about is your stance on types as first class citizens in Dokimos. Looking at the Datamodel, Map<String, Object> is used at the core to hold data.

This approach is similar to how the big eval libs such as https://deepeval.com operate, but I've started to wonder whether it would be a good idea to enforce types using generics more strictly. Data represented in a Map<String, Object> can always be represented in a complex object, as long as the "keys" are well known.

Using generics one could imagine types such as DataSet<INPUT,EXPECTED_OUTPUT> and Example<INPUT, EXPECTED_OUTPUT>. Test cases could look like EvalTestCase<INPUT, EXPECTED_OUTPUT. ACTUAL_OUTPUT>.

This would allow using Typed Evaluators, which operate on concrete fields of INPUT, EXPECTED_OUTPUT and ACTUAL_OUTPUT, which enables reuse of Evaluators without having to deconstruct the Task result into specific Map Entries beforehand and knowing the keys where the data for Evaluators lie.

I think such an approach would play to the strengths of the JMV. Have you thought about something like this? Where do you see the drawbacks?

[fkapsahili]

Hi @stefanks-px, thanks a lot for opening this, and it's great to hear that you like the project and that you're using Dokimos on your own evaluation workflows!

The background of the quite loose data model on some of the core objects inside dokimos-core, like input, expectedOutput, and metadata, is really that I wanted it to be as flexible as possible in the early days of the project. I also wanted it to feel idiomatic to people coming from the Python eval libraries, so a lot of the concepts are really close to what you find in the Python ecosystem, and picking up Dokimos would not mean learning a new mental model first.

Over time I figured that it'd probably make sense to have the freedom to be stricter on the typing side of things where you want it, which is also what led to typed outputs. Task.typed() lets a task return a real domain object instead of a hand-assembled map, and actualOutputAs(Movie.class), expectedOutputAs() and inputAs() read it back, with OutputType<T> for generic targets, such as List<Movie> for example, because Class<T> cannot express those otherwise.

The Kotlin side also has these equivalents, e.g. typedTask<T> { }, actualOutputAs<T>(). There is also a docs page on Structured & Typed Data and the per-method reference on the datamodel docs page. So a custom evaluator can already work on concrete fields today.

What you describe here is a great input and goes further than what we have today! And the reasons I have not gone to Dataset<INPUT, EXPECTED_OUTPUT> are mostly that Datasets are loaded at runtime from JSON/CSV, and a type parameter there would be a claim and cannot be guaranteed and this would fail at runtime if there's a mismatch. I also think that three parameters do not fully cover the evaluator surface, which needs to be quite flexible. For example, RAG evaluators want usually a retrieval context next to the answer, the agent evaluator wants tool calls, tool defs, and tasks, which also sit next to the output, so something like EvalTestCase<I, E, A> either keeps a map for the rest or keeps growing possible parameters. This would also make evaluators less reusable; today one ExactMatchEvaluator instance can run against any experiment. If we'd have something like Evaluator<I, E, A>, this would only fit matching ones, and for other evaluators we'd need to bridge them with adapters or variable tricks.

I could absolutely think of something like a typed base class, like TypedEvaluator<T> that does the conversion for you and hands you evaluate(T actual, T expected). This would give you the field-level access without trying to make everything generic. What do you think? Would something like that cover what you're missing / looking for? :)

[stefanks-px]

Thanks for your fast reply!

I haven't seen a complete RAG eval example yet but isn't the ContextualRelevanceEvaluator also just looking up a concrete key in the actual output map? If the Task is responsible for storing values under the retrievalContextKey then a structured Object could be a replacement. Instead of the key you would need to supply a Getter to retrieve the context object from the actual output object.

While the ExactMatchEvaluator is so generic that it can be used in all Experiments it's usefulness is limited, if structured output with types is the primary operation mode.

Our Task outputs often look like this:

public record Output(
        String title,
        String summary
        Category category
) {}

Different fields require different Evaluators but at the same time it would be clunky to deconstruct the Object into the actual output map. Applying the ExactMatchEvaluator to category right now would require the definition of a new Evaluator to extract a category from the input (golden) and the actual output and then delegating to the ExactMatchEvaluator.

We provided our own TypedEvaluator which looks like this

public class TypedEvaluator<ACTUAL, EXPECTED, MAPPED> implements Evaluator {

    private final Class<ACTUAL> actualOutputClass;
    private final Class<EXPECTED> expectedOutputClass;
    private final Function<ACTUAL, MAPPED> actualMapper;
    private final Function<EXPECTED, MAPPED> expectedMapper;
    private final Evaluator evaluator;

which works fine but is also kind of clunky to use.

I guess what I am looking for is a clear separation between evaluation logic, which is already provided by Dokimos in the Evaluators, and a means to unpack fields from objects for the evaluation.

Using generics could help here but would be a clear shift from the current data model. In order words you would bind Evaluators to a concrete Experiment<I, E, A> and provide the mapping functions instead of keys.

[fkapsahili]

@stefanks-px I think we've converged to the same thing :) You're right that the keys are really just untyped getters (including the retrievalContextKey), so the separation you're describing is a feature where Evaluators keep the scoring logic, and an optional projection layer does the unpacking. Task.typed() made the write-side typed and the *As() accessors made the read-side typed, but on the evaluator-side of things you still end up hand-wiring what you built. I think we could close that loop with a first-class version of your TypedEvaluator:

Evaluator category = TypedEvaluator.of(Output.class)
   .name("Category")
   .extracting(Output::category)
   .evaluateWith(ExactMatchEvaluator.builder().build());

This would read both sides through the existing typed accessors, so the single class token also covers the golden, which is still raw JSON at runtime. That's why I'd keep the token-pattern and not push the generics into Dataset or Experiment. The golden side can't guarantee a typed evaluator anyway, and this was every existing evaluator would still work unchanged underneath.

Something I'd really like to understand a bit better: what's the clunky part in your version, the class tokens or the wiring at the call site?

I'd open a proposal issue with an API sketch and tag you there :)

[stefanks-px]

@fkapsahili Sorry for the late reply

Does Output.class refer to actualOutputClass or expectedOutputClass in your example? They might not be the same e.g. a AI agent produces are complex object but your expectedOutputClass just holds some ground truths or whether results should be found.

TypedEvaluator could provide a simple API where both classes are from the same type and a more complex API if the two classes differ.

All in all I think we are on the right track :) . Would gladly review your proposal.

[fkapsahili]

@stefanks-px No worries :) And that's a good question!

Output.class is the actual output type. The golden gets its own token whenever it needs one, so both of the variants you're describing exist:

// same shape on both sides, the extractor just gets mirrored onto the golden
TypedEvaluator.of(Output.class)
        .name("Category")
        .extracting(Output::category)
        .evaluateWith(ExactMatchEvaluator.builder().build());

// golden only holds the ground truths, so it brings its own type and getter
TypedEvaluator.of(Output.class)
        .name("Title")
        .extracting(Output::title)
        .expecting(Golden.class, Golden::headline)
        .evaluateWith(ExactMatchEvaluator.builder().build());

Your agent case is the second one, where the actual side is the complex object and the golden is just whatever subset you actually want to assert on.

I wrote the whole thing up in #180 with the full surface and the reasoning behind it, including a couple of things I'm genuinely unsure about (whether name(...) should be mandatory, and what should happen when the golden can't be converted). Would be great if you had a look.

And I'm still curious about the other thing: what was the clunky part in your version, the class tokens or the wiring at the call site? That's the bit I want to get right.

[stefanks-px]

@fkapsahili Mainly the class tokens but I don't see a way around this right now. Looking into your write up now.