Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented Mar 12, 2016

JIRA: https://issues.apache.org/jira/browse/SPARK-13839

What changes were proposed in this pull request?

This PR improves Filter codegen by:

  1. After [SPARK-13751] [SQL] generate better code for Filter #11585, we filter out the rows early if it have null value. So we can only evaluate the inputs used in these filtering before this plan. The evaluation of other inputs can be deferred after the filtering done.
  2. Currently we check IsNotNull(a) to find the attributes needed to filter out null values. However, the expression IsNotNull(Cast(a, dt)) will not be checked by this. We should add pattern to find this.

How was this patch tested?

Existing tests.

@SparkQA
Copy link

SparkQA commented Mar 12, 2016

Test build #52997 has finished for PR 11676 at commit b905863.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 12, 2016

Test build #52998 has finished for PR 11676 at commit c539777.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya viirya force-pushed the improve-filter-codegen branch from c539777 to 6e4748b Compare March 12, 2016 12:39
@SparkQA
Copy link

SparkQA commented Mar 12, 2016

Test build #53003 has finished for PR 11676 at commit 6e4748b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya viirya changed the title [SPARK-XXX][SQL] Defer input evaluation and fix Cast issue in IsNotNull filtering for Filter codegen [SPARK-13839][SQL] Defer input evaluation and fix Cast issue in IsNotNull filtering for Filter codegen Mar 13, 2016
@viirya
Copy link
Member Author

viirya commented Mar 17, 2016

cc @davies Can you please review this?

case IsNotNull(a) if child.output.contains(a) => true
case IsNotNull(a) =>
a match {
case Casts(a) if child.output.contains(a) => true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not add these corner cases here, they should be handled by constraints.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I will send other PR for this.

@viirya viirya closed this Mar 17, 2016
@viirya viirya deleted the improve-filter-codegen branch December 27, 2023 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants