Behavior for external path #823
Conversation
benc-db
requested review from
andrefurlan-db and
rcypher-databricks
as code owners
| @@ -112,6 +112,7 @@ class DatabricksConfig(AdapterConfig): | |||
| file_format: str = "delta" | |||
| table_format: TableFormat = TableFormat.DEFAULT | |||
| location_root: Optional[str] = None | |||
| include_full_name_in_path: bool = False | |||
There was a problem hiding this comment.
can it be moved to the last argument to be backward compatible if someone calls w/o named arguments
There was a problem hiding this comment.
Normally I would agree with you, however...
Configs are never invoked with named arguments, as they are not actually exposed to users. Instead, users provide config via yaml, which dbt bundles together into a generic config object, which gets cast for type-checking purposes. This process is generic, i.e. dbt is also not invoking DatabricksConfig with named arguments. The primary reason to add fields here is to get code completion support, since basically all config provided by user (through yaml file) ends up available on the config object, and if you add the field here, then your IDE knows it will be on the object. Given that, it is more valuable to keep config fields that are related close together than to worry about ordering.
Resolves #812
Description
Adding config to have dbt-databricks add catalog and schema information to the path of external tables, to reduce the likelihood of conflicts arising from tables with the same name. This is a huge pain in testing, but not sure if others hit this in the wild, hence why default is to not do this behavior. We probably should have always been doing this, but it could potentially break existing workflows, another reason to default to False.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.