[FEATURE]: Integration with Data Quality Monitoring - Phase 1

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

DQ at Databricks feels fragmented: DQX, DQM, DLT expectations

Proposed Solution

Provide unification. As part of Phase 1:

• Integrate with Data Quality Monitoring (DQM in Unity Catalog) health indicators: https://docs.databricks.com/aws/en/data-quality-monitoring/anomaly-detection/#-view-data-quality-monitoring-results-in-the-ui
• DQX results available in quality: https://docs.databricks.com/aws/en/admin/system-tables/data-quality-monitoring
• DQX available in DQM Visualizations

Additional Context

No response

[IlyasFardaouix]

I think the key challenge here is integrating the DQX results with DQM Visualizations, which seems to be the most complex part of Phase 1. I'm curious, do you have any insight into how the existing DQX results are being fetched and processed in the databricks_labs_dqx_app.backend.app module, as that might give us a clue on how to unify the data with DQM?