Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-42500][SQL] ConstantPropagation support more case
### What changes were proposed in this pull request? This PR enhances ConstantPropagation to support more cases. Propagated through other binary comparisons. Propagated across equality comparisons. This can be further optimized to false. ### Why are the changes needed? Improve query performance. [Denodo](https://community.denodo.com/docs/html/browse/latest/en/vdp/administration/optimizing_queries/automatic_simplification_of_queries/removing_redundant_branches_of_queries_partitioned_unions) also has a similar optimization. For example: ``` CREATE TABLE t1(a int, b int) using parquet; CREATE TABLE t2(x int, y int) using parquet; CREATE TEMP VIEW v1 AS SELECT * FROM t1 JOIN t2 WHERE a = x AND a = 0 UNION ALL SELECT * FROM t1 JOIN t2 WHERE a = x AND (a IS NULL OR a <> 0); SELECT * FROM v1 WHERE x > 1; ``` Before this PR: ``` == Optimized Logical Plan == Union false, false :- Project [a#0 AS a#12, b#1 AS b#13, x#2 AS x#14, y#3 AS y#15] : +- Join Inner : :- Filter (isnotnull(a#0) AND (a#0 = 0)) : : +- Relation spark_catalog.default.t1[a#0,b#1] parquet : +- Filter (isnotnull(x#2) AND ((0 = x#2) AND (x#2 > 1))) : +- Relation spark_catalog.default.t2[x#2,y#3] parquet +- Join Inner, (a#16 = x#18) :- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16))) : +- Relation spark_catalog.default.t1[a#16,b#17] parquet +- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0))) +- Relation spark_catalog.default.t2[x#18,y#19] parquet ``` After this PR: ``` == Optimized Logical Plan == Join Inner, (a#16 = x#18) :- Filter ((isnull(a#16) OR NOT (a#16 = 0)) AND ((a#16 > 1) AND isnotnull(a#16))) : +- Relation spark_catalog.default.t1[a#16,b#17] parquet +- Filter ((isnotnull(x#18) AND (x#18 > 1)) AND (isnull(x#18) OR NOT (x#18 = 0))) +- Relation spark_catalog.default.t2[x#18,y#19] parquet ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. Closes apache#42038 from TongWei1105/SPARK-42500. Authored-by: TongWei1105 <vvtwow@gmail.com> Signed-off-by: Yuming Wang <yumwang@ebay.com>
- Loading branch information