Fix some issues with non-serializable inputs in evals #1333
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmontagu
commented
Apr 1, 2025
| @@ -83,6 +83,8 @@ def _stringify(value: Any) -> str: | |||
| if isinstance(value, str): | |||
| return value | |||
| try: | |||
| # If the value can be serialized to JSON, use that. | |||
| # If that behavior is undesirable, the user could manually call repr on the arguments to the judge_* functions | |||
| return to_json(value).decode() | |||
There was a problem hiding this comment.
I at first thought that we should remove the JSON serialization here too, and I'm still not confident we shouldn't, but you do have the option to call repr on the inputs before passing them to the judge_ functions. Admittedly, you'd need to make a custom evaluator to do that, but 🤷♂️ I'm not convinced this is wrong yet. Would be happy to change it to just use repr rather than json serializing if others feel strongly that that's better though.
Docs Preview
|
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I realized I shouldn't have been eagerly dumping to JSON, I used to do that because I wanted inputs to be a
dict[str, Any]but there was never really a benefit to that and we did change the default type for inputs toAny. And it just isn't necessary for the rendering. So I think this is better.Closes #1325
I think this addresses the main issue reported in https://pydanticlogfire.slack.com/archives/C083V7PMHHA/p1743485529924179