Skip to content

Conversation

@ahibrahimm
Copy link
Contributor

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@ahibrahimm ahibrahimm requested a review from a team as a code owner November 14, 2025 14:47
Copilot AI review requested due to automatic review settings November 14, 2025 14:47
@github-actions github-actions bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Nov 14, 2025
Copilot finished reviewing on behalf of ahibrahimm November 14, 2025 14:50
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request adds functionality to handle string inputs and single-entry lists in the GroundednessEvaluator. The changes improve input validation by extracting context validation logic into a separate method and adding checks for single-entry inputs.

Key Changes:

  • Extracted context validation logic into a reusable _validate_context method
  • Added _is_single_entry helper method to detect single-entry inputs (strings or single-item lists)
  • Implemented validation to raise NOT_APPLICABLE error when no context is available for single-entry query/response pairs
  • Modified response filtering to conditionally call _filter_file_search_results only when context is valid

target=ErrorTarget.GROUNDEDNESS_EVALUATOR,
)

filtered_response = self._filter_file_search_results(response) if self._validate_context(context) else response
Copy link

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a potential type mismatch bug in this conditional logic. The _filter_file_search_results method expects a List[Dict[str, Any]] parameter, but response could be a string.

Looking at the code flow:

  1. Line 304 gets response from kwargs, which could be a string (per the first __call__ overload)
  2. This code path is only executed when context or conversation is NOT provided (line 301-302)
  3. If _validate_context(context) returns True, it calls _filter_file_search_results(response)
  4. However, _filter_file_search_results at line 332 expects messages: List[Dict[str, Any]]

If response is a string and _validate_context(context) is True, this will cause a runtime error when _filter_file_search_results tries to iterate over the string and call .get() on characters.

Consider adding a type check before calling _filter_file_search_results or ensuring that this code path is only reached when response is guaranteed to be a list.

Suggested change
filtered_response = self._filter_file_search_results(response) if self._validate_context(context) else response
if self._validate_context(context) and isinstance(response, list):
filtered_response = self._filter_file_search_results(response)
else:
filtered_response = response

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants