feat(api): implement upsert() using MERGE INTO

[deepyaman]

Description of changes

Implement Backend.upsert() using sqlglot.expressions.merge() under the hood. Upsert support is very important, especially for data engineering use cases.

Starting with a most basic implementation, including only supporting one join column. I think this could be expanded to support a list without much effort.

MERGE INTO support is limited. DuckDB only added support for MERGE statements earlier today in 1.4.0, and many other backends don't support it. However, it seems like the more standard/correct approach for supporting upserts, and it doesn't require merge keys defined ahead of time on tables.

Backends that work:

• DuckDB (from 1.4.0)
• Flink
• Oracle (currently using a hack to work around "AS" getting added to MERGE statement)
• MS SQL (currently throwing a ; onto the end of every statement 😅)
• Postgres

Should work, need help to test:

• Databricks
• Snowflake
• BigQuery

Backends that don't work:

• PySpark ("MERGE INTO TABLE is not supported temporarily.")
• Clickhouse
• DataFusion
• SQLite (supports the nonstandard UPSERT statement)
• Impala ("The MERGE statement is only supported for Iceberg tables.")
• MySQL
• Polars
• RisingWave
• Athena ("MERGE INTO is transactional and is supported only for Apache Iceberg tables in Athena engine version 3.")
• Trino ("connector does not support modifying table rows")

Issues closed

• Resolves Add Backend.upsert #5391

[deepyaman]

@cpcloud Requesting review for initial feedback while I try to improve backend support; I assume I could always mark a lot of them notyet if the approach is correct but some translations just need work.

[deepyaman]

This is obviously unacceptable, but will wait on tobymao/sqlglot#5910 to try and figure out what's the best way to work arounds this (unless somebody else has ideas).

[georgesittas]

Fixed here tobymao/sqlglot#5911, will release soon.

[deepyaman]

@georgesittas Tested locally with SQLGlot 27.16.3, and removing this hack works perfectly. Thank you!!

(So impressed by the 0-issue inbox on SQLGlot BTW.)

[deepyaman]

These if statements can before backend-specific variables, something like https://github.com/ibis-project/ibis/blob/main/ibis/backends/flink/__init__.py#L65-L66? I wanted to try and see what the prevailing pattern is before finalizing the default.

[deepyaman]

I'm honestly not sure how to better do this; I wasn't able to figure out how to add a semicolon to an expression in SQLGlot.

[deepyaman]

@cpcloud I'm temporarily cherry-picked the changes from #11636 and the updates to xfail/xpass two DuckDB tests in order to get DuckDB and Oracle tests working (need the newer DuckDB and SQLGlot releases). All of these changes are in the last 3 commits:

• 7fddcb9
• 0ca5379
• adc70e7

With that, all of the functionality to implement Backend.upsert() is working and ready for review. The remaining issues all also exist in #11637, so I won't duplicate solving them. Whenever you do merge #11637, I should be able to back out the last 3 commits and rebase this on top.