Skip to content

[SPARK-33278][SQL][FOLLOWUP] Improve OptimizeWindowFunctions to avoid transfer first to nth_value. #30419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 31 commits into from

Conversation

beliefer
Copy link
Contributor

@beliefer beliefer commented Nov 19, 2020

What changes were proposed in this pull request?

#30178 provided OptimizeWindowFunctions used to transfer first to nth_value.
If the window frame is UNBOUNDED PRECEDING AND CURRENT ROW or UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, nth_value has better performance than first.
But the OptimizeWindowFunctions need to exclude other window frame.

Why are the changes needed?

Improve OptimizeWindowFunctions to avoid transfer first to nth_value if the specified window frame isn't UNBOUNDED PRECEDING AND CURRENT ROW or UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Does this PR introduce any user-facing change?

'No'.

How was this patch tested?

Jenkins test.

beliefer and others added 30 commits June 19, 2020 10:36
@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35921/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35921/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35922/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35922/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Test build #131318 has finished for PR 30419 at commit 6923dd1.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Test build #131321 has finished for PR 30419 at commit f7d7e7f.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@beliefer
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35939/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35939/

@SparkQA
Copy link

SparkQA commented Nov 19, 2020

Test build #131335 has finished for PR 30419 at commit f7d7e7f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@beliefer
Copy link
Contributor Author

cc @cloud-fan

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in f83fcb1 Nov 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants