[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition #31857

wangyum · 2021-03-17T03:23:37Z

What changes were proposed in this pull request?

Improve PropagateEmptyRelation to support join with false condition. For example:

SELECT * FROM t1 LEFT JOIN t2 ON false

Before this pr:

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, false
   :- FileScan parquet default.t1[a#4L]
   +- BroadcastExchange IdentityBroadcastMode, [id=#40]
      +- FileScan parquet default.t2[b#5L]

After this pr:

== Physical Plan ==
*(1) Project [a#4L, null AS b#5L]
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[a#4L]

Why are the changes needed?

Avoid BroadcastNestedLoopJoin to improve query performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

wangyum · 2021-03-17T06:26:26Z

cc @cloud-fan

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala

SparkQA · 2021-03-19T09:45:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40832/

SparkQA · 2021-03-19T09:54:59Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40832/

SparkQA · 2021-03-19T13:41:10Z

Test build #136250 has finished for PR 31857 at commit 25689da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-03-19T13:55:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

@@ -1468,6 +1469,12 @@ object PushPredicateThroughJoin extends Rule[LogicalPlan] with PredicateHelper {
    case _ => false
  }

+  private def pushDownJoinConditions(conditions: Seq[Expression], plan: LogicalPlan) = {
+    conditions
+      .filterNot(_.semanticEquals(TrueLiteral)) // Push down true condition is useless.


shouldn't this be optimized by BooleanSimplification already? true And cond -> cond

It is another issue. Will be fixed by another PR.

SparkQA · 2021-03-19T15:42:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40841/

SparkQA · 2021-03-19T15:51:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40841/

# Conflicts: # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala

SparkQA · 2021-03-19T17:49:10Z

Test build #136259 has finished for PR 31857 at commit debb094.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-19T18:28:15Z

Test build #136263 has finished for PR 31857 at commit 3202590.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-19T18:50:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40845/

SparkQA · 2021-03-19T19:00:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40845/

wangyum · 2021-03-19T23:58:41Z

retest this please.

SparkQA · 2021-03-20T00:43:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40854/

SparkQA · 2021-03-20T00:52:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40854/

SparkQA · 2021-03-20T04:29:01Z

Test build #136273 has finished for PR 31857 at commit 3202590.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2021-03-20T14:58:21Z

Merged to master.

dongjoon-hyun

+1, late LGTM.
Thank you, @wangyum and @cloud-fan .

Push down the foldable predicate to both sides of Join

9ad1833

wangyum commented Mar 17, 2021

View reviewed changes

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala Outdated Show resolved Hide resolved

github-actions bot added the SQL label Mar 17, 2021

cloud-fan reviewed Mar 17, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Mar 17, 2021

View reviewed changes

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala Outdated Show resolved Hide resolved

wangyum added 4 commits March 17, 2021 20:34

fix

7997eee

Fix

5988d28

Fix

ee5b4eb

pushDownJoinConditions

9e913e9

cloud-fan reviewed Mar 18, 2021

View reviewed changes

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala Show resolved Hide resolved

fix

25689da

cloud-fan reviewed Mar 19, 2021

View reviewed changes

fix

debb094

wangyum changed the title ~~[SPARK-28220][SQL] Push down the foldable predicate to both sides of Join~~ [SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition Mar 19, 2021

cloud-fan approved these changes Mar 19, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into SPARK-28220

3202590

# Conflicts: # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala

wangyum closed this in 908318f Mar 20, 2021

wangyum deleted the SPARK-28220 branch March 20, 2021 14:58

dongjoon-hyun reviewed Mar 20, 2021

View reviewed changes

[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition #31857

[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition #31857

Uh oh!

Conversation

wangyum commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

wangyum commented Mar 17, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

cloud-fan Mar 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyum Mar 19, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

wangyum commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

wangyum commented Mar 20, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wangyum commented Mar 17, 2021 •

edited

Loading

cloud-fan Mar 19, 2021 •

edited

Loading