Skip to content

Azure SDK V2-Execution of concurrent Drift and Data quality monitors #34413

Open

Description

  • Package Name: azure-ai-ml Azure SDK V2:
  • Package Version: 1.12.1:
  • Operating System: Linux, running it on azure notebooks on Azure portal.:
  • Python Version: 3.10:

Describe the bug
We are required to execute 150 plus data quality and data drift monitors. There are multiple errors that occur on different Azure managed components. Few errors are mentioned below.

  1. rejected, too many sessions are being created.(error1 image uploaded)
  2. something took too long to load while initial spark context was being setup.(error2 image uploaded)
  3. MountPointCreationError(Some(1)).(error3 image uploaded)
  4. No module named 'azure.storage.filedatalake.(error4 image uploaded)
  5. Cant get attribute '_execute' on module <azureml.dataprep.api._dataframereader>. (error5 image uploaded)

To Reproduce
Steps to reproduce the behavior:

  1. schedule multiple drift and data quality monitors like 100 at a same time.
  2. errors will show up in random Azure managed components.

Expected behavior
I expected multiple monitors to be executed concurrently.

Screenshots
error1
error2
error3
error4
error5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

ClientThis issue points to a problem in the data-plane of the library.Machine LearningService AttentionWorkflow: This issue is responsible by Azure service team.bugThis issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions