fix(data-diff): restore Table Diff on Databricks, Unity Catalog and AzureSQL

[IceS2]

Describe your changes:

Fixes #

Table Diff has been broken on Databricks and Unity Catalog since two unrelated changes landed in late 2025, and unreachable from the UI on AzureSQL since the second of them.

The run never returned. DatabricksBaseTableParameter built the data-diff URL from getattr(service_connection_config, "token", ""). #23561 moved token under authType on both connection schemas and touched no data-diff file, so from that commit the default kicked in and the URL carried an empty password. data-diff maps that password to access_token; databricks-sql-connector drops a falsy one (client.py:274), auth selection falls through to DatabricksOAuthProvider, and a headless agent parks on a browser redirect that never arrives. Every tableDiff on either connector hung until it was killed by hand.

The test was not offered in the UI. #24706 introduced supportedServices on tableDiff.json. The filter treats an absent list as "all services" (CollectionDAO.java:9079-9081), so adding an allowlist that omits Databricks, UnityCatalog and AzureSQL revoked the test from three connectors that each ship a data-diff parameter setter.

Type of change:

• Bug fix

High-level design:

The connection is now described to data-diff as a dict built by the connection class, instead of a hand-spliced URL:

{driver, server_hostname, http_path, auth_method, <credential fields>}

get_data_diff_connection_dict lives in databricks/auth.py next to get_auth_config, and both DatabricksConnection and UnityCatalogConnection expose it through get_connection_dict(). All three authentication types are expressed in full — personal access token, Databricks OAuth M2M, and Azure AD service principal — so none is reduced to a partial credential. Every value is a plain string, because data-diff caches connections on json.dumps of the config; providers are constructed on its side (open-metadata/collate-data-diff#33), which is why this PR floors collate-data-diff>=0.11.15.

A service-level dict cannot carry the catalog, and data-diff resolves a two-part schema.table path against the connection's catalog (defaulting to hive_metastore). DatabricksBaseTableParameter therefore injects the catalog and schema of each table, following the shape TrinoTableParameter and AzureSQLTableParameter already use. This also makes a cross-catalog diff on one service resolve each side independently.

The URL form was kept in an earlier draft and rejected: it cannot express Azure AD at all, and string-splicing credentials into an authority is what produced the original bug.

base_diff_params_setter now copies the connection dict before rewriting its driver — both sides of a same-service diff are handed the same object, so in-place mutation would let one side's scope overwrite the other's.

Rollout:

Blocked on the collate-data-diff 0.11.15 release; draft until then. The break has been present since 2025-10-03, so this likely warrants a backport to the active release lines as well as main.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 1986913722f7e1c42bb5bcc643870920129b8041 in Playwright run 31540101643, attempt 1.

✅ 613 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 26m 58s

⏱️ Max setup 3m 14s · max shard execution 17m 58s · max shard-job elapsed before upload 21m 17s · reporting 5s

🌐 217.27 requests/attempt · 2.75 app boots/UI scenario · 5.29% common-shard skew

Optimization targets still in progress:

• Browser traffic was 217.27 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.75 per UI scenario (1771 boots / 643 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	137	0	0	0	0	0
✅ Shard chromium-02	129	0	0	0	0	0
✅ Shard chromium-03	133	0	0	0	0	0
✅ Shard data-asset-rules-01	61	0	0	0	0	0
✅ Shard domain-isolation-01	14	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard ingestion-01	42	0	0	0	0	0
✅ Shard ingestion-02	22	0	0	0	0	0
✅ Shard reindex-01	2	0	0	0	0	0
✅ Shard search-01	10	0	0	0	0	0
✅ Shard search-rbac-01	29	0	0	0	0	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[gitar-bot]

Code Review ✅ Approved

Restores Table Diff on Databricks, Unity Catalog, and AzureSQL by reading connection credentials from authType and updating supported services. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}