feat(ingestion/unity): ingest Lakeflow pipeline expectations as assertions by acrylJonny · Pull Request #19755 · datahub-project/datahub

acrylJonny · 2026-09-12T17:31:48Z

Stacked on #19754 (Databricks data quality monitors) — review and merge that first.

Summary

Adds ingestion of Lakeflow Declarative Pipelines (formerly Delta Live Tables) expectation results as DataHub dataset assertions, in the Unity Catalog connector.

When pipeline_expectations.enabled: true, the connector lists the workspace's pipelines, reads each pipeline's event log, and for the most recent update emits a dataset-scoped assertion (plus a per-run result) for every expectation defined on a dataset. An expectation passes when no records failed it.

This is a pull-based sibling to the data-quality-monitor assertions added previously — a different source (pipeline event log vs. monitor profile-metrics table), so it lives in its own extractor.

Design notes

No SQL warehouse required. The event log is read over the pipelines REST API (GET /api/2.0/pipelines/{id}/events) rather than the SQL event_log() table-valued function. The raw REST endpoint is used because the typed SDK PipelineEvent model drops the details payload that carries the expectation metrics, and because the SQL TVF is owner-only.
Dataset resolution. Each expectation's short dataset name is resolved to a Unity Catalog dataset using the pipeline's target catalog and schema. Pipelines without a UC target are skipped and reported.
Assertion naming & category. Each expectation gets its own assertion type Databricks Pipeline Expectation and is named for the expectation (e.g. Rows meet expectation amount_positive), so expectations are distinguishable in the assertions list.
Action severity. The expectation's action is captured as a custom property and mapped to failure severity — expect_or_fail / expect_or_drop violations are recorded at high severity, a plain expect (warn-only) at low.
Idempotent identity. Assertion URNs are derived from the dataset URN, pipeline id, and expectation name (not the update), so re-ingesting accrues new run events rather than creating duplicates.
Degrades gracefully. A pipeline whose event log can't be read (e.g. missing permission) is reported as a warning and skipped, not fatal.
Disabled by default behind pipeline_expectations.enabled.

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Tests added (unit tests for the extractor + a golden-file integration test)
Docs updated (Unity Catalog connector docs)

Summary by cubic

Previously, Lakeflow pipeline expectations were not ingested; now the Unity Catalog connector publishes them as dataset assertions, with expectation dataset names (including fully-qualified and backtick-quoted identifiers) resolved correctly instead of being re-prefixed with the pipeline target.

Disabled by default behind pipeline_expectations.enabled; pipeline_pattern optionally filters which pipelines to read, and the catalog/schema/table patterns also apply per dataset so no assertions land on excluded tables.
Reads each pipeline's event log over the REST API, so no SQL warehouse is required and extraction still runs if the warehouse can't start; only the newest update is aggregated, and pagination stops once it's consumed.
Emits a dataset-scoped assertion (structured like dbt row-level native tests, with the expectation name as native type) plus a per-run result per expectation; an expectation passes when no records failed it, and expect_or_fail/expect_or_drop failures are high severity while plain expect is low.
Assertion URNs derive from dataset, pipeline, and expectation name, so re-ingestion accrues new run events instead of duplicates; dataset URNs respect include_metastore.
Pipelines without a Unity Catalog target, or whose target or event log can't be read, are skipped and reported as warnings, not fatal.
Ingestion principals need CAN_VIEW on pipelines whose expectations should be published.

^{Written for commit a939387. Summary will update on new commits.}

cursor · 2026-09-12T17:32:00Z

PR Summary

Overview
Adds optional ingestion of Lakeflow pipeline expectation results as DataHub dataset assertions in the Unity Catalog connector (pipeline_expectations.enabled, optional pipeline_pattern).

When enabled, the source lists pipelines, reads each pipeline’s event log over the REST API (no SQL warehouse), aggregates only the latest update (summing micro-batch flow_progress metrics), resolves expectation datasets to Unity Catalog tables (including FQ names and backtick-quoted identifiers), and emits idempotent assertion info + run events per expectation. Pass/fail is based on failed_records == 0, with FAIL/DROP vs ALLOW mapped to high/low failure severity. Pipelines without a UC target, inaccessible event logs, or datasets filtered by catalog/schema/table patterns are skipped with warnings/report counters.

Docs cover config, behavior, and CAN_VIEW permissions; proxy gains pipeline list/target/events helpers; unit and golden integration tests validate emitted MCPs.

^{Reviewed by Cursor Bugbot for commit a939387. Bugbot is set up for automated code reviews on this repo. Configure here.}

codecov · 2026-09-12T17:36:37Z

Codecov Report

❌ Patch coverage is 79.49791% with 49 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...estion/src/datahub/ingestion/source/unity/proxy.py	17.39%	19 Missing ⚠️
...ub/ingestion/source/unity/pipeline_expectations.py	90.13%	15 Missing ⚠️
...stion/src/datahub/ingestion/source/unity/source.py	16.66%	15 Missing ⚠️

📢 Thoughts on this report? Let us know!

cursor

Stale Bugbot comment from a previous run.

cubic-dev-ai

All reported issues were addressed

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 3 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

datahub-connector-tests · 2026-09-12T18:22:11Z

Connector Tests Results

All connector tests passed for commit a939387

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

cubic-dev-ai

All reported issues were addressed across 6 files (changes from recent commits).

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

…tions Publishes Lakeflow Declarative Pipelines (formerly Delta Live Tables) expectation results as DataHub dataset assertions. The pipeline event log is read over the REST API (no SQL warehouse required); for each pipeline's latest update, an expectation's passed/failed record counts become a dataset-scoped assertion plus a per-run result. Disabled by default behind pipeline_expectations.enabled. Co-authored-by: Cursor <cursoragent@cursor.com>

Lakeflow reports the expectation's dataset as catalog.schema.table on Unity Catalog pipelines, not a bare table name. The extractor was always prepending the pipeline target catalog/schema, producing a broken 5-part dataset URN. Parse the qualified name and fall back to the pipeline target only for missing components. Verified end-to-end against a live pipeline run. Co-authored-by: Cursor <cursoragent@cursor.com>

Emit Lakeflow pipeline expectations as structured custom assertions (scope/operator/aggregation/nativeType) mirroring the dbt connector's row-level native test, instead of a hand-written "Pipeline expectation" description. DataHub now renders "Dataset rows are passing assertion <expectation>", groups under the "Databricks" provider, and carries the expectation name as the native check type. Co-authored-by: Cursor <cursoragent@cursor.com>

Address review feedback on the Lakeflow expectations extractor: - Per-pipeline target lookup failures are caught and reported, so one bad pipeline no longer aborts extraction for the rest. - Read the pipeline event log to page-token exhaustion via a lazy generator, stopping once the newest update is consumed. Removes the fixed 20-page cap that could silently truncate events and undercount failures. - Include the metastore id in the assertion's dataset URN when include_metastore is enabled, so it resolves to the published dataset. - Split dataset identifiers on unquoted dots (handles backtick-quoted parts). - Extract expectations before the warehouse-gated profiling block, so the REST-only path still runs when the SQL warehouse fails to start. - Add tests for schema-qualified/quoted/metastore URN resolution, per-pipeline target errors, and graceful degradation on malformed event payloads. Co-authored-by: Cursor <cursoragent@cursor.com>

Give each Lakeflow (DLT) expectation its own custom assertion type ("Databricks Pipeline Expectation") and carry the expectation name in the description so it is distinguishable in the assertions list (the list renders a custom assertion's name from its description, not the structured native fields). Surface the expectation's action (ALLOW/DROP/FAIL) as a custom property and map it to failure severity: expect_or_fail / expect_or_drop are hard failures (HIGH), a plain expect only warns (LOW). Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

…erns Pipeline expectations are discovered from the event log independently of table ingestion, so a pipeline could reference datasets the recipe excluded (or intermediate pipeline views), producing assertions on entities that were never ingested. Apply the connector's catalog/schema/table filters before emitting, mirroring the table-ingestion path, and count skipped datasets in the report. Also make the metastore URN test non-vacuous: the fake dataset-URN builder now mirrors production gen_dataset_urn (str(ref)), so the test fails if the extractor stops threading the metastore through. Drop remaining other-connector references from assertion comments. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a939387. Configure here.}

cursor · 2026-09-13T01:23:33Z

+            self.config.catalog_pattern.allowed(catalog_id)
+            and self.config.schema_pattern.allowed(schema_id)
+            and self.config.table_pattern.allowed(ref.qualified_table_name)
+        )


Incomplete dataset ingest filters

Medium Severity

_pipeline_dataset_allowed only reapplies catalog_pattern, schema_pattern, and table_pattern. It ignores the catalogs allowlist that actually selects which catalogs are ingested, plus include_views / view_pattern / include_tables. Lakeflow datasets are often materialized views, so expectations can be published for entities this run never ingested.

^{Reviewed by Cursor Bugbot for commit a939387. Configure here.}

github-actions Bot added the ingestion PR or Issue related to the ingestion of metadata label Sep 12, 2026

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 17:33 View deployment

cursor Bot reviewed Sep 12, 2026

View reviewed changes

Comment thread metadata-ingestion/src/datahub/ingestion/source/unity/pipeline_expectations.py

Comment thread metadata-ingestion/src/datahub/ingestion/source/unity/proxy.py Outdated

Comment thread metadata-ingestion/src/datahub/ingestion/source/unity/pipeline_expectations.py

cubic-dev-ai Bot reviewed Sep 12, 2026

View reviewed changes

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 17:41 View deployment

cubic-dev-ai Bot reviewed Sep 12, 2026

View reviewed changes

acrylJonny force-pushed the databricks-pipeline-expectations branch from eb861a3 to 18634a9 Compare September 12, 2026 17:53

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 17:55 View deployment

vercel Bot deployed to Preview September 12, 2026 18:08 View deployment

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 19:39 View deployment

cubic-dev-ai Bot reviewed Sep 12, 2026

View reviewed changes

Comment thread metadata-ingestion/tests/unit/test_unity_catalog_pipeline_expectations.py

vercel Bot deployed to Preview September 12, 2026 19:52 View deployment

acrylJonny force-pushed the databricks-pipeline-expectations branch from 43483d4 to b658a62 Compare September 12, 2026 19:54

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 19:57 View deployment

cursor Bot reviewed Sep 12, 2026

View reviewed changes

Comment thread metadata-ingestion/src/datahub/ingestion/source/unity/pipeline_expectations.py

maggiehays added the needs-review Label for PRs that need review from a maintainer. label Sep 12, 2026

vercel Bot deployed to Preview September 12, 2026 20:08 View deployment

acrylJonny force-pushed the databricks-pipeline-expectations branch from b658a62 to b4c9ca9 Compare September 12, 2026 20:41

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 20:43 View deployment

cursor Bot reviewed Sep 12, 2026

View reviewed changes

Comment thread metadata-ingestion/src/datahub/ingestion/source/unity/pipeline_expectations.py

vercel Bot deployed to Preview September 12, 2026 20:56 View deployment

acrylJonny force-pushed the databricks-pipeline-expectations branch from b4c9ca9 to 7d805d3 Compare September 12, 2026 22:24

github-actions Bot deployed to datahub-wheels (Preview) September 12, 2026 22:26 View deployment

vercel Bot deployed to Preview September 12, 2026 22:41 View deployment

github-actions Bot deployed to datahub-wheels (Preview) September 13, 2026 01:07 View deployment

acrylJonny and others added 2 commits September 13, 2026 02:14

acrylJonny and others added 5 commits September 13, 2026 02:14

docs(unity): note pipeline expectation action severity

fd29d01

Co-authored-by: Cursor <cursoragent@cursor.com>

acrylJonny force-pushed the databricks-pipeline-expectations branch from 0de79a6 to a939387 Compare September 13, 2026 01:17

github-actions Bot deployed to datahub-wheels (Preview) September 13, 2026 01:19 View deployment

cursor Bot reviewed Sep 13, 2026

View reviewed changes

vercel Bot deployed to Preview September 13, 2026 01:35 View deployment

Uh oh!

Conversation

acrylJonny commented Sep 12, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design notes

Checklist

Summary by cubic

Uh oh!

cursor Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

codecov Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datahub-connector-tests Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Connector Tests Results

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 13, 2026

Choose a reason for hiding this comment

Incomplete dataset ingest filters

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

acrylJonny commented Sep 12, 2026 •

edited by cubic-dev-ai Bot

Loading

cursor Bot commented Sep 12, 2026 •

edited

Loading

codecov Bot commented Sep 12, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

datahub-connector-tests Bot commented Sep 12, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading