-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Handle the input data for pure string_check evaluation in result converter #43906
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR enhances the _convert_results_to_aoai_evaluation_results function to support input keys without the "inputs." prefix. Previously, only keys prefixed with "inputs." or "outputs." were processed; now non-prefixed keys are also included in the evaluation results.
Key changes:
- Modified the key processing logic to handle non-prefixed input keys
- Updated test data to include examples with non-prefixed keys (e.g.,
"query") - Updated expected test outputs to reflect the new behavior
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py | Changed from elif to else to process all non-output keys, with conditional prefix stripping for "inputs." prefixed keys |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/data/evaluation_util_convert_old_output_test.jsonl | Added "query" as a top-level key in test data to demonstrate support for non-prefixed input keys |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/data/evaluation_util_convert_expected_output.json | Updated expected output to include the new "query" field in rows and datasource_item objects |
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py
Show resolved
Hide resolved
|
/check-enforcer override |
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines