Skip to content

Commit

Permalink
Add exception for conversation + q/a and add breaking change warning …
Browse files Browse the repository at this point in the history
…to changelog
  • Loading branch information
diondrapeck committed Sep 4, 2024
1 parent 52c8756 commit 7c7f7f5
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 0 deletions.
3 changes: 3 additions & 0 deletions src/promptflow-evals/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
### Bugs Fixed
- Fixed evaluators to accept (non-Azure) Open AI Configs.

### Breaking Changes
- Replaced `jailbreak` parameter in `AdversarialSimulator` with `_jailbreak_type` parameter to support multiple jailbreak types. Instead of editing this parameter directly, we recommend using the `JailbreakAdversarialSimulator` class for UPIA jailbreak and `IndirectAttackSimulator` class for XPIA jailbreak.

### Improvements
- Set the PF_EVALS_BATCH_USE_ASYNC environment variable to True by default to enable asynchronous batch run for async-enabled built-in evaluators, improving performance.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ def __call__(
:return: The evaluation scores and reasoning.
:rtype: dict
"""
if conversation and any([question, answer]):
raise ValueError("'conversation' cannot be provided if 'question' and 'answer' are also provided.")

if conversation:
self._validate_conversation(conversation)

Expand Down

0 comments on commit 7c7f7f5

Please sign in to comment.