Support for Selecting Specific Paths from JSON Objects in Evaluation Prompts

[divyanshubhuta]

Describe the feature or potential improvement

Currently, the evaluation prompts only support using the entire metadata field from a JSON object, without the ability to select specific paths within the object. It would be beneficial to enhance this functionality by allowing users to specify and extract particular paths or keys from the JSON object for more targeted evaluation. This feature would improve precision and flexibility when working with complex metadata structures.

Additional information

No response

[marcklingen]

thanks for sharing, this is also interesting to extract from structured input/output fields as well

[marcklingen]

Additional context on this in #3352 by @stepanogil

Describe the feature or potential improvement

This is how you currently map variables in the model-based Evaluation (Beta) module:

i.e. you map whole returns of Input or Output or Metadata to a variable e.g. {{query}}

This is an example of how I populate the a Trace Input:

This means when I do Evals, this whole json blob is passed on as the value of my {{query}}:

Which I don't want to do as all the metadata will just introduce a lot of noise to the eval.
What I want to do is map my {{query}} to particular kv pair inside of the Input. Example below:

So I want another dropdown here where I can configure to which key I want to map the {{query}} variable.
So I can input something like: Input["conversation_history"][0]["content"] or something to this effect.