Skip to content

databricks: removes cluster on create table test and allows only partition #3191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 14, 2025
+9 −3
Merged

databricks: removes cluster on create table test and allows only partition #3191

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions tests/load/test_job_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ def test_complete_load(naming: str, client: SqlJobClientBase) -> None:

@pytest.mark.parametrize(
"client",
destinations_configs(default_sql_configs=True),
destinations_configs(default_sql_configs=True, exclude=["bigquery"]),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why exclude bigquery? I though this test was failing only with destination_type = databricks?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a bigquery variant below so this one is just repeating one of the cases of test_schema_update_create_table_bigquery_hidden_dataset. why a separate test for bigquery: you can have hidden datasets in bigquery. in that case we use very different code path to get schema of the dataset. they are not available in INFORMATION SCHEMA.

indirect=True,
ids=lambda x: x.name,
)
Expand All @@ -267,6 +267,9 @@ def test_schema_update_create_table(client: SqlJobClientBase) -> None:
# this will be destkey
sender_id = item_normalizer._infer_column("sender_id", "982398490809324")
assert sender_id["cluster"] is True
# disable cluster on databricks: it does not support PARTITION and CLUSTER on a single table
has_cluster = client.config.destination_type != "databricks"
sender_id["cluster"] = has_cluster
# this will be not null
record_hash = item_normalizer._infer_column("_dlt_id", "m,i0392903jdlkasjdlk")
assert record_hash["unique"] is True
Expand All @@ -276,8 +279,9 @@ def test_schema_update_create_table(client: SqlJobClientBase) -> None:
# check hints in schema update
table_update = schema_update[table_name]["columns"]
assert table_update["timestamp"]["sort"] is True
assert table_update["sender_id"]["cluster"] is True
assert table_update["sender_id"]["cluster"] is has_cluster
assert table_update["_dlt_id"]["unique"] is True
assert table_update["_dlt_id"]["nullable"] is False
_, storage_columns = list(client.get_storage_tables([table_name]))[0]
assert len(storage_columns) > 0

Expand All @@ -289,7 +293,9 @@ def test_schema_update_create_table(client: SqlJobClientBase) -> None:
ids=lambda x: x.name,
)
@pytest.mark.parametrize("dataset_name", (None, "_hidden_ds"))
def test_schema_update_create_table_bigquery(client: SqlJobClientBase, dataset_name: str) -> None:
def test_schema_update_create_table_bigquery_hidden_dataset(
client: SqlJobClientBase, dataset_name: str
) -> None:
# patch dataset name
if dataset_name:
# drop existing dataset
Expand Down
Loading