[SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates #3784

liancheng · 2014-12-24T05:56:59Z

This PR is a simplified version of several filter optimization rules introduced in #3778 authored by @scwf. Newly introduced optimizations include:

a && a => a
a || a => a
(a || b || c || ...) && (a || b || d || ...) => a && b && (c || d || ...)

The 3rd rule is particularly useful for optimizing the following query, which is planned into a cartesian product

SELECT *
  FROM t1, t2
 WHERE (t1.key = t2.key AND t1.value > 10)
    OR (t1.key = t2.key AND t2.value < 20)

to the following one, which is planned into an equi-join:

SELECT *
  FROM t1, t2
 WHERE t1.key = t2.key
   AND (t1.value > 10 OR t2.value < 20)

The example above is quite artificial, but common predicates are likely to appear in real life complex queries (like the one mentioned in #3778).

A difference between this PR and #3778 is that these optimizations are not limited to Filter, but are generalized to all logical plan nodes. Thanks to @scwf for bringing up these optimizations, and @chenghao-intel for the generalization suggestion.

SparkQA · 2014-12-24T05:57:30Z

Test build #24762 has started for PR 3784 at commit cf95639.

This patch merges cleanly.

SparkQA · 2014-12-24T05:57:34Z

Test build #24762 has finished for PR 3784 at commit cf95639.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T05:57:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24762/
Test FAILed.

SparkQA · 2014-12-24T06:12:25Z

Test build #24763 has started for PR 3784 at commit 2abbf8e.

This patch merges cleanly.

SparkQA · 2014-12-24T06:17:25Z

Test build #24764 has started for PR 3784 at commit 5d54349.

This patch merges cleanly.

SparkQA · 2014-12-24T06:24:05Z

Test build #24763 has finished for PR 3784 at commit 2abbf8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T06:24:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24763/
Test FAILed.

SparkQA · 2014-12-24T06:30:26Z

Test build #24764 has finished for PR 3784 at commit 5d54349.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T06:30:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24764/
Test FAILed.

SparkQA · 2014-12-24T07:47:29Z

Test build #24776 has started for PR 3784 at commit 3cf7937.

This patch merges cleanly.

SparkQA · 2014-12-24T07:52:28Z

Test build #24777 has started for PR 3784 at commit 0e51101.

This patch merges cleanly.

SparkQA · 2014-12-24T08:07:29Z

Test build #24779 has started for PR 3784 at commit 4ab3a58.

This patch merges cleanly.

SparkQA · 2014-12-24T08:59:40Z

Test build #24776 has finished for PR 3784 at commit 3cf7937.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T08:59:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24776/
Test PASSed.

SparkQA · 2014-12-24T09:04:58Z

Test build #24777 has finished for PR 3784 at commit 0e51101.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T09:05:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24777/
Test PASSed.

SparkQA · 2014-12-24T09:20:04Z

Test build #24779 has finished for PR 3784 at commit 4ab3a58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T09:20:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24779/
Test PASSed.

chenghao-intel · 2014-12-25T05:09:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

f.transformExpressionUp instead? And this rule also can be applied to the normal expressions, not just only for expression in Filter. Move this into BooleanSimplification?

Ah, f.transformExpressionUp is really a good idea! Moving to BooleanSimplification also makes sense. Thanks for the suggestions :)

chenghao-intel · 2014-12-25T05:14:45Z

This is really helpful optimization, and the implementation makes sense to me, and it would be great if we apply the rules for the those non Filter expressions also.

SparkQA · 2014-12-25T09:27:29Z

Test build #24815 has started for PR 3784 at commit 00aaa63.

This patch merges cleanly.

SparkQA · 2014-12-25T09:32:31Z

Test build #24816 has started for PR 3784 at commit f4d9b8f.

This patch merges cleanly.

SparkQA · 2014-12-25T09:37:30Z

Test build #24817 has started for PR 3784 at commit 601d5f6.

This patch merges cleanly.

SparkQA · 2014-12-25T10:38:28Z

Test build #24815 has finished for PR 3784 at commit 00aaa63.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-25T10:38:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24815/
Test PASSed.

SparkQA · 2014-12-25T10:45:32Z

Test build #24816 has finished for PR 3784 at commit f4d9b8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-25T10:45:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24816/
Test PASSed.

SparkQA · 2014-12-25T10:49:04Z

Test build #24817 has finished for PR 3784 at commit 601d5f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-25T10:49:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24817/
Test PASSed.

scwf · 2014-12-25T14:56:53Z

Hi @liancheng, i admit my PR is more complicated, but this only cover three cases, i think we'd better adding a separate rule to optimize And/Or in sql for as many as possible cases, not mix several cases in BooleanSimplification. So I am refactorying my PR to make it more readable and clean.

SparkQA · 2014-12-25T15:47:30Z

Test build #24821 has started for PR 3784 at commit caca560.

This patch merges cleanly.

SparkQA · 2014-12-25T16:58:12Z

Test build #24821 has finished for PR 3784 at commit caca560.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-25T16:58:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24821/
Test PASSed.

scwf · 2014-12-26T01:41:57Z

Hi, @liancheng, my PR originally also not limited to Filter, i used transformExpressionsDown from my first version, the tittle of my first version is not accurate:)

liancheng · 2014-12-27T05:35:04Z

Hey @scwf, I've posted my reply in #3778, so lets discuss these rules there to prevent distraction.

marmbrus · 2014-12-30T21:38:45Z

Thanks! I've merged this to master.

Adds an optimization rule for filter normalization

cf95639

liancheng mentioned this pull request Dec 24, 2014

[SPARK-4937][SQL] Adding optimization to simplify the And, Or condition in spark sql #3778

Closed

liancheng changed the title ~~[SPARK-4937] Normalizes conjunctions and disjunctions to eliminate common predicates~~ [SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates Dec 24, 2014

liancheng added 2 commits December 24, 2014 14:08

Forgot our sacred Apache licence header...

2abbf8e

Fixes typo in comment

5d54349

liancheng force-pushed the normalize-filters branch from 3cf7937 to 0e51101 Compare December 24, 2014 07:52

Fixes test failure, adds more tests

4ab3a58

liancheng force-pushed the normalize-filters branch from 0e51101 to 4ab3a58 Compare December 24, 2014 08:04

chenghao-intel reviewed Dec 25, 2014
View reviewed changes

liancheng force-pushed the normalize-filters branch from 00aaa63 to f4d9b8f Compare December 25, 2014 09:31

liancheng force-pushed the normalize-filters branch from f4d9b8f to 601d5f6 Compare December 25, 2014 09:33

Moves filter normalization into BooleanSimplification rule

caca560

liancheng force-pushed the normalize-filters branch from 601d5f6 to caca560 Compare December 25, 2014 15:44

asfgit closed this in 61a99f6 Dec 30, 2014

liancheng deleted the normalize-filters branch December 31, 2014 00:29

liancheng mentioned this pull request Dec 23, 2015

[SPARK-6624][SQL]Add CNF Normalization as part of optimization #8200

Closed

[SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates #3784

[SPARK-4937][SQL] Normalizes conjunctions and disjunctions to eliminate common predicates #3784

Uh oh!

Conversation

liancheng commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 24, 2014

Uh oh!

AmplabJenkins commented Dec 24, 2014

Uh oh!

chenghao-intel Dec 25, 2014

Choose a reason for hiding this comment

Uh oh!

liancheng Dec 25, 2014

Choose a reason for hiding this comment

Uh oh!

chenghao-intel commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

AmplabJenkins commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

AmplabJenkins commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

AmplabJenkins commented Dec 25, 2014

Uh oh!

scwf commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!

AmplabJenkins commented Dec 25, 2014

Uh oh!

scwf commented Dec 26, 2014

Uh oh!

liancheng commented Dec 27, 2014

Uh oh!