-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-32577][SQL][TEST] Fix the config value for shuffled hash join in test in-joins.sql #33236
Conversation
cc @cloud-fan, thanks. |
Kubernetes integration test starting |
Kubernetes integration test status success |
Test build #140720 has finished for PR 33236 at commit
|
are there more places that need to fix? |
It seems not searchable in repo, let me try to figure out all places tomorrow. Thanks. |
let me just merge it in then. |
…in test in-joins.sql ### What changes were proposed in this pull request? We found the `in-join.sql` does not test shuffled hash join properly in https://issues.apache.org/jira/browse/SPARK-32577, but didn't find a good way to fix it. Given we now have a test config to enforce shuffled hash join in #33182, we can fix the test here now as well. ### Why are the changes needed? Fix test to have better test coverage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Reran the test to compare the output, and verified the query plan manually to make sure shuffled hash join being used. Closes #33236 from c21/join-test. Authored-by: Cheng Su <chengsu@fb.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit f3c1159) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Merged to master and branch-3.2. |
Thank you @HyukjinKwon and @cloud-fan for review! Will submit another PR for other changes. |
…ash join for all other test queries ### What changes were proposed in this pull request? This is the followup from #33236 (comment), where we are fixing the config value of shuffled hash join, for all other test queries. Found all configs by searching in https://github.com/apache/spark/search?q=spark.sql.join.preferSortMergeJoin . ### Why are the changes needed? Fix test to have better test coverage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closes #33249 from c21/join-test. Authored-by: Cheng Su <chengsu@fb.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…ash join for all other test queries ### What changes were proposed in this pull request? This is the followup from #33236 (comment), where we are fixing the config value of shuffled hash join, for all other test queries. Found all configs by searching in https://github.com/apache/spark/search?q=spark.sql.join.preferSortMergeJoin . ### Why are the changes needed? Fix test to have better test coverage. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closes #33249 from c21/join-test. Authored-by: Cheng Su <chengsu@fb.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 23943e5) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
What changes were proposed in this pull request?
We found the
in-join.sql
does not test shuffled hash join properly in https://issues.apache.org/jira/browse/SPARK-32577, but didn't find a good way to fix it. Given we now have a test config to enforce shuffled hash join in #33182, we can fix the test here now as well.Why are the changes needed?
Fix test to have better test coverage.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Reran the test to compare the output, and verified the query plan manually to make sure shuffled hash join being used.