[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction #29800

beliefer · 2020-09-18T07:50:47Z

What changes were proposed in this pull request?

Spark SQL supports some window function like NTH_VALUE.
If we specify window frame like UNBOUNDED PRECEDING AND CURRENT ROW or UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, we can elimate some calculations.
For example: if we execute the SQL show below:

SELECT NTH_VALUE(col,
         2) OVER(ORDER BY rank UNBOUNDED PRECEDING
        AND CURRENT ROW)
FROM tab;

The output for row number greater than 1, return the fixed value. otherwise, return null. So we just calculate the value once and notice whether the row number less than 2.
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING is simpler.

Why are the changes needed?

Improve the performance for NTH_VALUE, FIRST_VALUE and LAST_VALUE.

Does this PR introduce any user-facing change?

'No'.

How was this patch tested?

Jenkins test.

SparkQA · 2020-10-27T07:36:18Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34920/

SparkQA · 2020-10-27T08:52:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34924/

SparkQA · 2020-10-27T09:13:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34926/

SparkQA · 2020-10-27T09:16:52Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34924/

SparkQA · 2020-10-27T09:42:20Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34926/

SparkQA · 2020-10-27T10:12:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34927/

SparkQA · 2020-10-27T10:15:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34929/

SparkQA · 2020-10-27T10:36:24Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34929/

SparkQA · 2020-10-27T10:39:25Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34927/

SparkQA · 2020-10-27T12:37:24Z

Test build #130322 has finished for PR 29800 at commit ca574d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-27T13:05:38Z

Test build #130324 has finished for PR 29800 at commit fd59f6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-27T14:02:01Z

Test build #130326 has finished for PR 29800 at commit 9c35ddd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-27T14:02:46Z

Test build #130325 has finished for PR 29800 at commit 72e7805.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-10-27T14:34:16Z

sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala

+ *     1. [[FrameLessOffsetWindowFunction]] returns the value of the input column offset by a number
+ *        of rows according to the current row.
+ *     2. [[UnboundedOffsetWindowFunctionFrame]] and [[UnboundedPrecedingOffsetWindowFunctionFrame]]
+ *       returns the value of the input column offset by a number of rows within the partition.


within the partition -> within the frame

cloud-fan · 2020-10-27T14:37:28Z

sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala

+      if (inputIterator.hasNext) inputIterator.next()
+      inputIndex += 1
+    }
+    if (inputIndex >= 0 && inputIndex < input.length) {


inputIndex >= 0 seems always true

cloud-fan · 2020-10-27T14:39:03Z

sql/core/src/test/resources/sql-tests/results/window.sql.out

+-- !query output
+Larry Bott	11798	NULL
+Gerard Bondur	11472	Gerard Bondur
+Pamela Castillo	11303	Gerard Bondur


Not related to this PR. We should fix the test framework so that the result is always aligned.

The output comes from hiveResultString

SparkQA · 2020-10-28T03:08:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34950/

SparkQA · 2020-10-28T03:34:32Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34950/

SparkQA · 2020-10-28T06:33:51Z

Test build #130348 has finished for PR 29800 at commit 1c0e82b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-10-28T06:40:20Z

thanks, merging to master!

beliefer · 2020-10-28T07:13:03Z

@cloud-fan Thanks for your help!

### What changes were proposed in this pull request? #29800 provides a performance improvement for `NTH_VALUE`. `FIRST_VALUE` also could use the `UnboundedOffsetWindowFunctionFrame` and `UnboundedPrecedingOffsetWindowFunctionFrame`. ### Why are the changes needed? Improve the performance for `FIRST_VALUE`. ### Does this PR introduce _any_ user-facing change? 'No'. ### How was this patch tested? Jenkins test. Closes #30178 from beliefer/SPARK-33278. Lead-authored-by: gengjiaan <gengjiaan@360.cn> Co-authored-by: beliefer <beliefer@163.com> Co-authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

beliefer and others added 30 commits June 19, 2020 10:36

Reuse completeNextStageWithFetchFailure

4a6f903

Merge remote-tracking branch 'upstream/master'

96456e2

Merge remote-tracking branch 'upstream/master'

4314005

Merge remote-tracking branch 'upstream/master'

d6af4a7

Merge remote-tracking branch 'upstream/master'

f69094f

Merge remote-tracking branch 'upstream/master'

b86a42d

Merge branch 'master' of github.com:beliefer/spark

2ac5159

Merge remote-tracking branch 'upstream/master'

9021d6c

Merge branch 'master' of github.com:beliefer/spark

74a2ef4

Merge remote-tracking branch 'upstream/master'

9828158

Merge remote-tracking branch 'upstream/master'

9cd1aaf

Merge remote-tracking branch 'upstream/master'

abfcbb9

Merge remote-tracking branch 'upstream/master'

07c6c81

Support window function nth_value

c9a96c3

Merge remote-tracking branch 'upstream/master'

580130b

Remove some code.

bc0c308

Merge branch 'support-nth_value' of github.com:beliefer/spark

d95a7b7

Update golden file.

addcdbc

Merge branch 'master' of github.com:beliefer/spark

3712808

Merge remote-tracking branch 'upstream/master'

6107413

Fix conflict

2aee591

Merge remote-tracking branch 'upstream/master'

4b799b4

Try to optimize nth_value

a9c17a4

Optimize code and add test cases.

8778412

Merge branch 'support-nth_value' into optimize-nth_value

f190dd8

Optimize code.

db2b1d4

Merge branch 'support-nth_value' into optimize-nth_value

7ff1815

Fix bug.

5cb4686

Update golden file.

97f6376

Update golden file.

4002aaf

beliefer added 2 commits October 27, 2020 16:04

Optimize code

ca574d9

Optimize code

fd59f6e

beliefer added 2 commits October 27, 2020 17:20

Optimize code

72e7805

Optimize code

9c35ddd

cloud-fan reviewed Oct 27, 2020

View reviewed changes

cloud-fan approved these changes Oct 27, 2020

View reviewed changes

Optimize code

1c0e82b

cloud-fan closed this in 3c3ad5f Oct 28, 2020

beliefer mentioned this pull request Oct 29, 2020

[SPARK-33278][SQL] Improve the performance for FIRST_VALUE #30178

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction #29800

[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction #29800

beliefer commented Sep 18, 2020 •

edited

Loading

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

cloud-fan Oct 27, 2020

beliefer Oct 28, 2020

cloud-fan Oct 27, 2020

beliefer Oct 28, 2020

cloud-fan Oct 27, 2020

beliefer Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

cloud-fan commented Oct 28, 2020

beliefer commented Oct 28, 2020

[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction #29800

[SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction #29800

Conversation

beliefer commented Sep 18, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

cloud-fan Oct 27, 2020

Choose a reason for hiding this comment

beliefer Oct 28, 2020

Choose a reason for hiding this comment

cloud-fan Oct 27, 2020

Choose a reason for hiding this comment

beliefer Oct 28, 2020

Choose a reason for hiding this comment

cloud-fan Oct 27, 2020

Choose a reason for hiding this comment

beliefer Oct 28, 2020

Choose a reason for hiding this comment

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

SparkQA commented Oct 28, 2020

cloud-fan commented Oct 28, 2020

beliefer commented Oct 28, 2020

beliefer commented Sep 18, 2020 •

edited

Loading