Skip to content

[SPARK-26599][SQL]BroardCast hint can not work with PruneFileSourcePartitions #23517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

eatoncys
Copy link
Contributor

@eatoncys eatoncys commented Jan 11, 2019

What changes were proposed in this pull request?

BroardCast hint can not work with PruneFileSourcePartitions, for example, when the filter condition p is a partition field, table b in SQL below cannot be broadcast.

sql("select /*+ broadcastjoin(b) */ * from (select a from empty_test where p=1) a " + "join (select a,b from par_1 where p=1) b on a.a=b.a").explain

The reason is that PruneFileSourcePartitions deleted ResolvedHint when matching PhysicalOperations, the plan after PruneFileSourcePartitions before this PR is:

image

The original reason for this problem is that when PhysicalOperation matches ResolvedHint, it returns directly to its child nodes and skips over itself.
case h: ResolvedHint => collectProjectsAndFilters(h.child)

This PR adds processing to ResolvedHint , when matched to it in PruneFileSourcePartitions, skip it directly and parse its child nodes.

How was this patch tested?

Unit test of PruneFileSourcePartitionsSuite

@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101068 has finished for PR 23517 at commit f7bca2f.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@eatoncys
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101077 has finished for PR 23517 at commit 78e19b5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@eatoncys
Copy link
Contributor Author

It has been resolved by another pr #23507

@eatoncys eatoncys closed this Jan 11, 2019
@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101071 has finished for PR 23517 at commit f7bca2f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants