Description
openedon Sep 18, 2024
Is your feature request related to a problem? Please describe.
The azure-ai-evaluation
SDK exposes a credential
field in its evaluators (example: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/_content_safety/_violence.py#L14) to be used when calling the Azure AI RAI service. Right now, this credential is optional, and at execution time we choose DefaultAzureCredential
if the user did not pass anything in. However, we need to make this credential
parameter required in the evaluators since choosing a default credential is not safe SDK behavior.
If the evaluator is being run on a single input OR when it is used in the batch evaluate
API with no credential being passed in, the customer will not see any errors. However, if the user passes a credential to the evaluator and uses it in the evaluate
function, it will not work since evaluate
uses multiprocessing to do batch runs of multiple evaluators. Multiprocessing requires the evaluator to be pickle-able, but the credentials in azure-identity
are not pickle-able, which breaks this scenario.
Describe the solution you'd like
The credentials in azure-identity
should be pickle-able so that they can be used in evaluators when doing batch evaluation using evaluate
.
Metadata
Assignees
Labels
Type
Projects
Status
Done