Skip to content

Base engine adapter's merge method does not support WHEN NOT MATCHED BY SOURCE in Databricks #5355

@sjdur0

Description

@sjdur0

The merge method in the base EngineAdapter class always appends a WHEN NOT MATCHED THEN INSERT * after all other match statements - the code in the snippet below always executes.

        # This block always executes in the `EngineAdapter.merge` method
        match_expressions.append(
            exp.When(
                matched=False,
                source=False,
                then=exp.Insert(
                    this=exp.Tuple(expressions=[exp.column(col) for col in columns_to_types]),
                    expression=exp.Tuple(
                        expressions=[
                            exp.column(col, MERGE_SOURCE_ALIAS) for col in columns_to_types
                        ]
                    ),
                ),
            )
        )

This breaks any merges using the WHEN NOT MATCHED BY SOURCE ... clauses supported by Databricks since the strict order of merge clauses is:

  1. WHEN MATCHED
  2. WHEN NOT MATCHED [BY TARGET]
  3. WHEN NOT MATCHED BY SOURCE

Any deviation from this order results in a syntax error.

I'm happy to contribute to a solution, but some guidance on where is best to make the change would be ideal. I'd guess either defining a custom merge method for the DatabricksEngineAdapter or adding a check prior to appending the WHEN NOT MATCHED THEN INSERT *.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingEngine: DatabricksIssues related to Databricks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions