Allow clickable links as labels for test cases in evals experiments

[indigosquirrelio]

Description

This is either a feature request or a documentation request—not completely sure which, as I just haven't figured out how to do it.

We'd like to use the logfire web UI to view dataset runs, but it doesn't seem trivial to attach a label with clickable links. You can imagine there's other useful views for each eval case we run that could live outside of the logfire ecosystem (managed by us).

Without this, I'd probably want to view it elsewhere and then link back to logfire where applicable. If I'm missing something obvious, lmk!

References

No response

[DouweM]

@indigosquirrelio Are you looking to attach a static link to a specific case or dataset, or a dynamic link for a given experiment (dataset run)? We're adding support for custom metadata on experiments: #3263. In the frontend, we could render those in a special way including making experiment fields that contain a URL into a link.

cc @dmontagu

[indigosquirrelio]

We'd like a dynamic link for each test case. You can imagine an experiment where we run an agent on a set of inputs, and each agent run is subsequently viewable in our own UI.

I attempted to workaround this by creating an evaluator that renders the URL in a label column on each test case, but it didn't seem to have any way to format the value as a link (since span links get rendered, I was hoping I could figure something out, but wasn't able to).

[dmontagu]

Okay, so I'm happy to brainstorm here, and we should probably talk with @petyosi (who has done a lot of the building of the evals tables etc.), but here's a (straw-man?) proposal: if you put a pydantic.networks.AnyUrl object (or an instance of one of its subclasses) as a value in the case metadata, we render it in the title cell for the case.

I guess that's a bit awkward because I think we already have a metadata column for the case, and it would make sense for metadata values to go there. But maybe we add a new str | None-valued field description on the case that is expected to be markdown and can make reference to the metadata (e.g. using something like mustache templating), and then we can render the case title + description in the case cell? Given it's markdown then we don't need to do anything funky with URL types, it can just use normal markdown URL syntax (and the URLs can be built referencing metadata parameters, or just hard-coded if that's easier/necessary). It would probably be easier to use a static case-level field but I can see how that would get repetitive in e.g. a serialized dataset...

In your case I guess it might make sense for this to be a dataset-level template and to populate it in a case-specific way based on the case metadata?

Any reaction to the above? If it's interesting to you I think we can make something like this work. Is there anything this wouldn't do for you or that you don't like? Any alternative approaches you'd prefer?

[indigosquirrelio]

Is there a reason that we can't expose some formatter for each column within the evals framework? I was sort of expecting this to exist.

But exposing a description field that renders markdown would accomplish everything that I need, as long as it can be defined during an eval run. Though it doesn't feel right, I'd be happy enough for the time being :)

As a side note, it would be cool to update https://github.com/pydantic/logfire/blob/main/docs/guides/web-ui/evals.md with some info about how the frontend rendering works and maybe some examples of how we can configure the columns that get shown from pydantic-evals.

For example, is it correct that I'm supposed to be creating a custom Evaluator in order to get a label added as a column? Should it be possible to add an attribute / metadata there (this is what I was also expecting, but didn't figure out how)?