[Databricks destination] Adding comment and tags for table and columns and applying primary and foreign key constraints in Unity Catalog

[bayees]

Description

Create comments and tags for tables and columns in Unity Catalog tables. Also adding primary key and foreign key constraints using the primary_key and references hints.

I have added the create_indexes to enable primary and foreign key to keep backwards compatible

There is a new Databricks adapter to use the new x-databricks-cluster, x-databricks-table-comment, x-databricks-table-tags, x-databricks-column-comment, x-databricks-column-tags hints.

The table and column comments using description hints if those are available, but overrides those if the hints from the databricks adapter i given.

Related Issues

• Closes [Databricks] Add comments and tags hints for table and column #2625

Additional Context

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	f1d93ab
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/6878c376dc24cb000847c079
😎 Deploy Preview	https://deploy-preview-2674--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[rudolfix]

@bayees this looks good! are you able to fix linting errors and add a basic test? like we do ie. for bigquery adapter. also we can take over this PR as we are allowed to push to your branch. ping us what's your plan

[bayees]

@rudolfix Linting is done.

I am not sure about the test. Can you point to the specific Bigquery test?

[rudolfix]

Hey! BigQuery tests are in tests/load/bigquery/test_biguery_table_builder.py. example test checking if partition is right,

def test_bigquery_partition_by_integer(
    destination_config: DestinationTestConfiguration,
) -> None:
    pipeline = destination_config.setup_pipeline(f"bigquery_{uniq_id()}", dev_mode=True)

    @dlt.resource(
        columns={"some_int": {"data_type": "bigint", "partition": True, "nullable": False}},
    )
    def demo_resource() -> Iterator[Dict[str, int]]:
        for i in range(10):
            yield {
                "some_int": i,
            }

    @dlt.source(max_table_nesting=0)
    def demo_source() -> DltResource:
        return demo_resource

    pipeline.run(demo_source())

    with pipeline.sql_client() as c:
        with c.execute_query(
            "SELECT EXISTS (SELECT 1 FROM INFORMATION_SCHEMA.PARTITIONS WHERE partition_id IS NOT"
            " NULL);"
        ) as cur:
            has_partitions = cur.fetchone()[0]
            assert isinstance(has_partitions, bool)
            assert has_partitions

note that:

    with pipeline.sql_client() as c:
        databricks = c.native_connection

gives you access to native databricks client

you could place your tests in tests/load/databricks/test_databricks_adapter.py. also we do not need tests that are as comprehensive as BigQuery but a basic check if hints are translated in the right props in Unity Catalog would be cool.

regarding credentials:
you can place them into tests/.dlt/secrets.toml tests will find it. here are ours (without secrets). I assume you have access to the cluster?

[destination.databricks.credentials]
server_hostname = "adb-8001321225760611.11.azuredatabricks.net"
http_path = "/sql/protocolv1/o/8001321225760611/0124-183359-ghw1vo3b"
access_token = "..."
catalog = "dlt_ci"
client_id = "..."
client_secret = "..."

one more thing: we broke our devel :/ so you'd need to merge it into your branch,

[bayees]

@rudolfix
I added the unit test to the new Databricks adapter, but I found a non-deterministic issue. When added foreign keys to the tables, the constraint requires the related table with the primary key to exist. Are there any method for controlling the other of table creation?

[rudolfix]

@bayees here's PR #2791 that allows to add statements after table create/alter. you should override _get_table_post_update_sql , and move generation of FOREIGN REFERENCES there. We'll merge this PR asap and then you can merge it into your branch.

In the meantime we updated our dependency system to uv and we have new test workflows on ci. So overall things are better and much faster but please take a look at our CONTRIBUTING guide. Thanks!

[bayees]

@rudolfix Updated the fork to reflect the changes in #2791. Waiting for it to be merge.

Nice with uv. 🥇

[burnash]

@bayees thanks so much. I've reviewed #2791, as soon as the CI pass we'll merge it and I'll review this PR

[bayees]

Hey @burnash. Thank you for the review. I have implemented the changes suggested.

Please take a look at it and let me know if there are additional changes needed

[burnash]

by using the adapter method table_comment.

@bayees could you clarify if you mean "adapter method table_comment" or "adapter parameter table_comment"?

[burnash]

Same here

[burnash]

Looks very good, @bayees, thank you for the documentation and tests. Have a couple of minor comments, please take a look.

[bayees]

This was the recommendation from @rudolfix. Do you have input here?

[burnash]

This was the recommendation from @rudolfix.

@bayees just to clarify, which of the comments you're referring to?

[nicor88]

Thanks @bayees for this PR - for what I understand, this PR doesn't cover the possibility of marking a column a partition, adding z-order for a column, or liquid clustering, this will be a relevant feature to add, which is pretty relevant for teams that care about "query optimization" and cost control.

I don't think that is a good idea to add the above feature in this PR, but might be relevant to track that in other issue or then have other code changes related to it, WDYT @burnash ?

[burnash]

@bayees please take a look at documentation comments as well. Let me know if you need help with fixing mypy on CI. We'd need to make sure the tests pass before we merge the PR.

[burnash]

@nicor88 thanks for the suggestion. Agreed, it's best to open a new related issue for this.

[burnash]

Hi @bayees @christian-bay-backstage, just following up to see if you'd like to continue working on this PR or if you'd prefer that I take it from here. Thanks again for your effort!

[bayees]

Hi @bayees @christian-bay-backstage, just following up to see if you'd like to continue working on this PR or if you'd prefer that I take it from here. Thanks again for your effort!

I am currently on vacation, so if the only things to change is the phrasing in the documentation and fix the mypy, then please feel free to fix. Otherwise I will look into it in a week or two.

[burnash]

@bayees thank you very much for the contribution.