[SPARK-51984][SQL] Support Check Constraint enforcement in UpdateTable and MergeIntoTable #50943
Conversation
| // applied before the rewrite rules. So we need to apply `ResolveEncodersInUDF` before the | ||
| // rewrite rules. | ||
| // applied before the rewrite rules. So we need to apply the rewrite rules after | ||
| // `ResolveEncodersInUDF` | ||
| Batch("DML rewrite", fixedPoint, |
There was a problem hiding this comment.
Context of why this batch is not in the main resolution batch: fab6d83
| // Combine the check invariants into a single expression using conjunctive AND. | ||
| checkInvariants.reduceOption(And).fold(v2Write)( | ||
| condition => v2Write.withNewQuery(Filter(condition, v2Write.query))) | ||
| case r: DataSourceV2Relation => |
There was a problem hiding this comment.
The changes in this file is purely code clean up. cc @aokolnychyi
|
I will improve the rule to exclude adding FILTER for DELETE and add tests. |
|
cc @aokolnychyi |
| @@ -419,6 +419,10 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor | |||
| ResolveRowLevelCommandAssignments :: | |||
| MoveParameterizedQueriesDown :: | |||
| BindParameters :: | |||
| ResolveEncodersInUDF :: | |||
There was a problem hiding this comment.
I am moving the rules in to the main resolution batch, so that UDF encoders and table constraints can be correctly handled in DML writes.
cc @viirya @Ngone51 @aokolnychyi
There was a problem hiding this comment.
Reverted this one. It ends up failing many tests: https://github.com/gengliangwang/spark/runs/42517140406
| case r: DataSourceV2Relation => | ||
| buildCheckCondition(r.table).map { condition => | ||
| val filter = Filter(condition, v2Write.query) | ||
| // Resolve attribute references in the filter condition only, not the entire query. |
There was a problem hiding this comment.
See https://github.com/gengliangwang/spark/actions/runs/15130563620/job/42530638818 for details.
=== Result of Batch DML rewrite ===
!'Sort [(cast(x#26 as bigint) + cast('c.y as bigint)) ASC NULLS FIRST], true Sort [(cast(x#26 as bigint) + cast(tempresolvedcolumn(c#24.y, c, y, false) as bigint)) ASC NULLS FIRST], true
+- Aggregate [c#24.x], [c#24.x AS x#26] +- Aggregate [c#24.x], [c#24.x AS x#26]
+- SubqueryAlias t +- SubqueryAlias t
+- LocalRelation [c#24] +- LocalRelation [c#24]
,{})
What changes were proposed in this pull request?
Why are the changes needed?
Support check constraint enforcement on table update/merge into.
Does this PR introduce any user-facing change?
No
How was this patch tested?
New UT
Was this patch authored or co-authored using generative AI tooling?
No