-
Notifications
You must be signed in to change notification settings - Fork 3.2k
[Groudnedness] Handle strings and single entries #44033
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This pull request adds functionality to handle string inputs and single-entry lists in the GroundednessEvaluator. The changes improve input validation by extracting context validation logic into a separate method and adding checks for single-entry inputs.
Key Changes:
- Extracted context validation logic into a reusable
_validate_contextmethod - Added
_is_single_entryhelper method to detect single-entry inputs (strings or single-item lists) - Implemented validation to raise
NOT_APPLICABLEerror when no context is available for single-entry query/response pairs - Modified response filtering to conditionally call
_filter_file_search_resultsonly when context is valid
...valuation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py
Show resolved
Hide resolved
...valuation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py
Show resolved
Hide resolved
| target=ErrorTarget.GROUNDEDNESS_EVALUATOR, | ||
| ) | ||
|
|
||
| filtered_response = self._filter_file_search_results(response) if self._validate_context(context) else response |
Copilot
AI
Nov 14, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a potential type mismatch bug in this conditional logic. The _filter_file_search_results method expects a List[Dict[str, Any]] parameter, but response could be a string.
Looking at the code flow:
- Line 304 gets
responsefrom kwargs, which could be a string (per the first__call__overload) - This code path is only executed when
contextorconversationis NOT provided (line 301-302) - If
_validate_context(context)returns True, it calls_filter_file_search_results(response) - However,
_filter_file_search_resultsat line 332 expectsmessages: List[Dict[str, Any]]
If response is a string and _validate_context(context) is True, this will cause a runtime error when _filter_file_search_results tries to iterate over the string and call .get() on characters.
Consider adding a type check before calling _filter_file_search_results or ensuring that this code path is only reached when response is guaranteed to be a list.
| filtered_response = self._filter_file_search_results(response) if self._validate_context(context) else response | |
| if self._validate_context(context) and isinstance(response, list): | |
| filtered_response = self._filter_file_search_results(response) | |
| else: | |
| filtered_response = response |
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines