Skip to content

[SPARK-24402] [SQL] Optimize In expression when only one element in the collection or collection is empty #21797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

dbtsai
Copy link
Member

@dbtsai dbtsai commented Jul 17, 2018

What changes were proposed in this pull request?

Two new rules in the logical plan optimizers are added.

  1. When there is only one element in the Collection, the
    physical plan will be optimized to EqualTo, so predicate
    pushdown can be used.
    profileDF.filter( $"profileID".isInCollection(Set(6))).explain(true)
    """
      |== Physical Plan ==
      |*(1) Project [profileID#0]
      |+- *(1) Filter (isnotnull(profileID#0) && (profileID#0 = 6))
      |   +- *(1) FileScan parquet [profileID#0] Batched: true, Format: Parquet,
      |     PartitionFilters: [],
      |     PushedFilters: [IsNotNull(profileID), EqualTo(profileID,6)],
      |     ReadSchema: struct<profileID:int>
    """.stripMargin
  1. When the Collection is empty, and the input is nullable, the
    logical plan will be simplified to
    profileDF.filter( $"profileID".isInCollection(Set())).explain(true)
    """
      |== Optimized Logical Plan ==
      |Filter if (isnull(profileID#0)) null else false
      |+- Relation[profileID#0] parquet
    """.stripMargin

TODO:

  1. For multiple conditions with numbers less than certain thresholds,
    we should still allow predicate pushdown.
  2. Optimize the In using tableswitch or lookupswitch
    when the numbers of the categories are low, and they are Int,
    Long.
  3. The default immutable hash trees set is slow for query, and we
    should do benchmark for using different set implementation for faster
    query.
  4. filter(if (condition) null else false) can be optimized to false.

How was this patch tested?

Couple new tests are added.

// TODO: `EqualTo` for structural types are not working. Until SPARK-24443 is addressed,
// TODO: we exclude them in this rule.
&& !v.isInstanceOf[CreateNamedStructLike]
&& !newList.head.isInstanceOf[CreateNamedStructLike]) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan @gatorsmile until #21470 is merged, let's exclude CreateNamedStructLike in this rule.

@SparkQA
Copy link

SparkQA commented Jul 17, 2018

Test build #93186 has finished for PR 21797 at commit 8bc7573.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 17, 2018

Test build #93192 has finished for PR 21797 at commit 79d14f1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dbtsai
Copy link
Member Author

dbtsai commented Jul 17, 2018

Test it again.

@SparkQA
Copy link

SparkQA commented Jul 18, 2018

Test build #93203 has finished for PR 21797 at commit 68e9f04.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 18, 2018

Test build #93202 has finished for PR 21797 at commit 2e62027.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dbtsai
Copy link
Member Author

dbtsai commented Jul 18, 2018

Merged into master as it passes the build now.

@asfgit asfgit closed this in 681845f Jul 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants