-
Notifications
You must be signed in to change notification settings - Fork 147
[FIX]: Fix issue with cloning of external tables #1095
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ericj-db
merged 8 commits into
databricks:main
from
samgans:fix_issue_with_external_tables
Jul 12, 2025
Merged
[FIX]: Fix issue with cloning of external tables #1095
ericj-db
merged 8 commits into
databricks:main
from
samgans:fix_issue_with_external_tables
Jul 12, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d8d846d
to
46dbcd5
Compare
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
… branch Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
46dbcd5
to
5239635
Compare
Signed-off-by: Anton Zhyltsou <antonzhiltsov2000@gmail.com>
0f76ed8
to
f379790
Compare
Signed the last commit. |
ericj-db
approved these changes
Jul 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR resolves the bugs related to the external cloning introduced in #1079 (sorry for introducing them 🥲).
What are these bugs?
The cloning of the external tables still doesn't work, because the
{% if other_existing_relation.is_external %}
branch isn't followed due to multiple issues:is_external_table
instead ofis_external
. This is the typo that was introduced by me.is_external_table
returnsFalse
for every external table due to the fact that theget_uc_tables_sql
macro casts all the tables that are of'EXTERNAL', 'MANAGED', 'MANAGED_SHALLOW_CLONE', 'EXTERNAL_SHALLOW_CLONE'
types to theTABLE
type thus making any external-relatedDatabricksRelationType
not available astype
attribute of theDatabricksRelation
.In addition to that, the
databricks__create_or_replace_clone_external
macro was not available for the usage due to the dispatch syntax used in its declaration. Instead, we should have declared it ascreate_or_replace_clone_external
.Bug 1
To fix the bug 1, I've simply fixed the call of the function.
File 1.
Bug 2
To fix the bug 2, I've changed the
get_uc_tables_sql
macro to also return an attribute nameddatabricks_table_type
which additionally classifies the databricks table. To make this work, I've created another enumerationDatabricksTableType
which means the specific databricks table type and made this as adatabricks_table_type
attribute of theDatabricksRelation
. The Relation uses this attribute to determine if it is an external table or not.File 1.
Adding
databricks_table_type
attribute instead of simply updating thetype
attribute was done to keep the backwards compatibility of the attribute and not break potential assumptions about this attribute.File 2.
Introducing the new attribute returned from the macro has also required me to fix the type definitions of the return rows, thus I've introduced the new named tuple
DatabricksRelationInfo
so that the return values of the query that describes the relation are better documented, at the same time keeping the backwards-compatible type.File 3.
Bug 3
The dispatch of the external macro was fixed by removing the dispatch-based syntax in its definition.
File 1.
Testing
We've found a way to test these changes in our workspace, so we were able to run the clone operation successfully using the external tables:
Checklist
CHANGELOG.md
and added information about my change to the "dbt-databricks next" section.