-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance regression after adding support for SMJ with join filter #901
Comments
Disabling sortMergeJoin via configs restores the original performance. |
Thanks @andygrove I'm planning to profile it. |
Hmm, @andygrove Can you verify it again? In unit test, we have a test for CometSortMergeJoinExec metrics. It should have SQL metrics. |
I ran again with latest from main (0033), and then with SMJ + join filter disabled manually (0034). Here are the event logs. |
Hmm, I will run locally to see why the metrics are not there but they are in unit test. |
Im running slightly changed Q72 in DF
HashJoin - 9sec, SMJ - 20 sec Having LEFT OUTER joins removed the results are still the same HashJoin - 8.5 SMJ - 19 sec I'll build a flamegraph for SMJ soon |
@andygrove I just ran a simple sort merge join query locally on Spark 4.0 + Comet built from latest Looks like the metrics are shown for CometSort / CometSortMergeJoin. |
Describe the bug
When running TPC-DS benchmarks against 100 GB data set I see a large regression in performance. For example, here are the timings for q72 before and after adding support for SMJ with join condition.
Adding support for SMJ with join condition means that more of the plan is likely running natively and the performance issue isn't necessarily directly related to SMJ.
before
after
A secondary issue is that I do not see metrics for CometSort / CometSortMergeJoin.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: