[SPARK-31390][SQL][DOCS] Document Window Function in SQL Syntax Section #28220

huaxingao · 2020-04-15T01:13:42Z

What changes were proposed in this pull request?

Document Window Function in SQL syntax

Why are the changes needed?

Make SQL Reference complete

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Manually build and check

SparkQA · 2020-04-15T01:24:57Z

Test build #121296 has finished for PR 28220 at commit 572f7d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-04-15T01:40:44Z

cc @maropu

docs/sql-ref-syntax-qry-window.md

maropu · 2020-04-15T09:58:05Z

also cc: @viirya

maropu · 2020-04-15T12:01:45Z

docs/sql-ref-syntax-qry-window.md

-**This page is under construction**
+### Description
+
+Similarly to aggregate functions, window functions operate on a group of rows. However, unlike aggregate functions, window functions perform aggregation without reducing, calculating a return value for each row in the group. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative, or accessing the value of rows given the relative position of the current row. Spark SQL supports three types of window functions:


How about Similarly to aggregate functions, window functions operate on a group of rows. -> A window function operates on a group of rows and this is comparable to aggregate functions. ?

window functions perform aggregation without reducing, calculating a return value for each row in the group. is not clear. This means window functions do not compute a single aggregated value. Instead, they can generate multiple aggregated values for each group?

SparkQA · 2020-04-15T16:56:18Z

Test build #121331 has finished for PR 28220 at commit 5fadfae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-04-15T18:37:08Z

docs/sql-ref-syntax-qry-window.md

-**This page is under construction**
+### Description
+
+Similarly to aggregate functions, window functions operate on a group of rows. However, unlike aggregate functions, window functions perform aggregation without reducing, calculating an aggregated value for each row in the specified window. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative, or accessing the value of rows given the relative position of the current row. Spark SQL supports three types of window functions:


"without reducing"? Sounds confusing. How about "without reducing the number of rows"?

And "but calculating an aggregated value for each row in the specified window."

SparkQA · 2020-04-15T19:08:46Z

Test build #121333 has finished for PR 28220 at commit 52922f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-04-15T19:16:24Z

docs/sql-ref-syntax-qry-window.md

+### Description
+
+Window functions operate on a group of rows, referred to as a window, and calculate an aggregated value for each row based on the specified window. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative, or accessing the value of rows given the relative position of the current row. Spark SQL supports three types of window functions:
+


Does this look better now? @maropu @viirya

Also cc @srowen
Please feel free to rephrase. Thanks!

computing a cumulative -> computing a cumulative sum (or anything similar: average, statistic)

Thanks, it looks better. How about putting the last statement in a new line?;

...the current row. Spark SQL supports three types of window functions: * Ranking Functions * Analytic Functions * Aggregate Functions

We need this list here? The Syntax section has the same list.

SparkQA · 2020-04-15T20:17:42Z

Test build #121337 has finished for PR 28220 at commit 3b0c0a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-16T00:21:00Z

docs/sql-ref-syntax-qry-window.md

+    Specifies a comma separated list of key and value pairs for partitions.<br><br>
+    <b>Syntax:</b><br>
+      <code>
+        { PARTITION | DISTRIBUTE } BY partition_col_name  = partition_col_val ( [ , ... ] )


nit: I found the double spaces in this line.

maropu · 2020-04-16T00:21:59Z

docs/sql-ref-syntax-qry-window.md

+          MAX | MIN | COUNT | SUM | AVG | ...
+        </code>
+        <br>
+        Please refer <a href="api/sql/">here</a> for a complete list of Spark Aggregate Functions.


nit: here -> the Built-in Function document?

nit: Spark Aggregate Functions. -> Spark aggregate functions.?

I will put sql-ref-functions-builtin.html as the link for Built-in Function document. It's broken now but will work after your PR is in.

Ur, could you revert the link back? I'm currently not sure that my PR is target at 3.0.

maropu · 2020-04-16T00:26:23Z

docs/sql-ref-syntax-qry-window.md

+### Description
+
+Window functions operate on a group of rows, referred to as a window, and calculate an aggregated value for each row based on the specified window. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative, or accessing the value of rows given the relative position of the current row. Spark SQL supports three types of window functions:
+


Thanks, it looks better. How about putting the last statement in a new line?;

...the current row. Spark SQL supports three types of window functions: * Ranking Functions * Analytic Functions * Aggregate Functions

SparkQA · 2020-04-16T00:34:52Z

Test build #121342 has finished for PR 28220 at commit ea8ee10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-16T01:21:23Z

Test build #121344 has finished for PR 28220 at commit 116f403.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-16T02:18:41Z

docs/sql-ref-syntax-qry-window.md

+  <dt><code><em>window_function</em></code></dt>
+  <dd>
+    <ul>
+      <li> Ranking Functions </li>


nit: <li> Ranking Functions </li> -> <li>Ranking Functions</li>

maropu · 2020-04-16T02:20:09Z

docs/sql-ref-syntax-qry-window.md

+          MAX | MIN | COUNT | SUM | AVG | ...
+        </code>
+        <br>
+        Please refer to the <a href="sql-ref-functions-builtin.html">Built-in Function</a> document for a complete list of Spark aggregate functions.


nit: Built-in Function -> Built-in Functions by referring to the title in the doc: https://spark.apache.org/docs/latest/api/sql/index.html

maropu · 2020-04-16T02:24:00Z

docs/sql-ref-syntax-qry-window.md

+    Specifies an ordering of the rows.<br><br>
+    <b>Syntax:</b><br>
+      <code>
+        { ORDER | SORT } BY { expression [ ASC | DESC ] [ NULLS { FIRST | LAST } ] [ , ... ] }


Could you move ORDER BY and PARTITION BY caluses into the Syntax section like the Pg doc one?

[ existing_window_name ] [ PARTITION BY expression [, ...] ] [ ORDER BY expression [ ASC | DESC | USING operator ] [ NULLS { FIRST | LAST } ] [, ...] ] [ frame_clause ]

https://www.postgresql.org/docs/current/sql-select.html

viirya · 2020-04-16T05:20:02Z

docs/sql-ref-syntax-qry-window.md

-**This page is under construction**
+### Description
+
+Window functions operate on a group of rows, referred to as a window, and calculate an aggregated value for each row based on the specified window. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative statistic, or accessing the value of rows given the relative position of the current row.


"... calculate a return value for each row based on a group of rows"

SparkQA · 2020-04-16T06:24:40Z

Test build #121352 has finished for PR 28220 at commit 6af2eba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-16T08:38:27Z

docs/sql-ref-syntax-qry-window.md

+          UNBOUNDED { PRECEDING | FOLLOWING }
+          | CURRENT ROW
+          | boolean_expression { PRECEDING | FOLLOWING }
+      </code> <br><br>


I think we need to describe what these clauses (RANGE, ROWS, BETWEEN, ...) are.

SparkQA · 2020-04-16T17:11:28Z

Test build #121371 has finished for PR 28220 at commit 3fb73f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-16T23:58:24Z

docs/sql-ref-syntax-qry-window.md

+### Examples
+
+{% highlight sql %}
+


nit: remove this blank.

maropu

Looks fine except for the existing two comments.

[SPARK-31390][SQL][DOCS] Document Window Function in SQL Syntax Section #28220 (comment)
[SPARK-31390][SQL][DOCS] Document Window Function in SQL Syntax Section #28220 (comment)

maropu · 2020-04-17T00:01:09Z

Could you update the screenshot in the description, too?

SparkQA · 2020-04-17T04:40:34Z

Test build #121393 has finished for PR 28220 at commit 747cfef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-04-17T18:08:57Z

@maropu I have addressed the last two comments and updated the screenshots in description. Thanks for reviewing!

huaxingao · 2020-04-17T18:09:33Z

cc @srowen for final sign off.

srowen · 2020-04-17T18:28:34Z

docs/sql-ref-syntax-qry-window.md

+  +-----+-----------+------+-----+
+
+SELECT name, salary,
+    LAG(salary) OVER (PARTITION BY dept ORDER BY salary) as lag,


as -> AS
but definitely don't change it just for that. Looks fine. I'll merge shortly

maropu · 2020-04-17T23:28:05Z

docs/sql-ref-syntax-qry-window.md

+  +-----+-----------+------+----------+
+
+SELECT name, dept, age, CUME_DIST() OVER (PARTITION BY dept ORDER BY age
+    RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as cume_dist FROM employees;


nit: as -> AS here, too.

I will fix this

SparkQA · 2020-04-18T00:14:14Z

Test build #121433 has finished for PR 28220 at commit 6a3d475.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? Document Window Function in SQL syntax ### Why are the changes needed? Make SQL Reference complete ### Does this PR introduce any user-facing change? Yes <img width="1050" alt="Screen Shot 2020-04-16 at 9 13 34 PM" src="https://user-images.githubusercontent.com/13592258/79531509-7bf5af00-8027-11ea-8291-a91b2e97a1b5.png"> <img width="1050" alt="Screen Shot 2020-04-16 at 9 14 12 PM" src="https://user-images.githubusercontent.com/13592258/79531514-7e580900-8027-11ea-8761-4c5a888c476f.png"> <img width="1050" alt="Screen Shot 2020-04-16 at 9 14 45 PM" src="https://user-images.githubusercontent.com/13592258/79531518-82842680-8027-11ea-876f-6375aa5b5ead.png"> <img width="1050" alt="Screen Shot 2020-04-16 at 9 15 10 PM" src="https://user-images.githubusercontent.com/13592258/79531521-844dea00-8027-11ea-8948-712f054d42ee.png"> <img width="1050" alt="Screen Shot 2020-04-16 at 9 15 25 PM" src="https://user-images.githubusercontent.com/13592258/79531528-8748da80-8027-11ea-9dae-a465286982ac.png"> ### How was this patch tested? Manually build and check Closes #28220 from huaxingao/sql-win-fun. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org> (cherry picked from commit 142f436) Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

maropu · 2020-04-18T00:32:33Z

Thanks! Merged to master/3.0.

huaxingao · 2020-04-18T00:36:23Z

Thanks, all!

probot-autolabeler bot added the DOCS label Apr 15, 2020

maropu mentioned this pull request Apr 15, 2020

[SPARK-31390][SQL][DOCS] Document Window Function #28157

Closed

maropu reviewed Apr 15, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Show resolved Hide resolved

maropu reviewed Apr 15, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Outdated Show resolved Hide resolved

maropu reviewed Apr 15, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Outdated Show resolved Hide resolved

maropu reviewed Apr 15, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Outdated Show resolved Hide resolved

maropu reviewed Apr 15, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Show resolved Hide resolved

maropu reviewed Apr 15, 2020

View reviewed changes

viirya reviewed Apr 15, 2020

View reviewed changes

huaxingao commented Apr 15, 2020

View reviewed changes

huaxingao added 4 commits April 15, 2020 17:07

[SPARK-31390][SQL][DOCS] Document Window Function in SQL syntax

5eee4ee

address comments

cc7c443

address comments

1c7f59f

address comments

ea8ee10

huaxingao force-pushed the sql-win-fun branch from 3b0c0a8 to ea8ee10 Compare April 16, 2020 00:13

maropu reviewed Apr 16, 2020

View reviewed changes

address comments

116f403

maropu reviewed Apr 16, 2020

View reviewed changes

viirya reviewed Apr 16, 2020

View reviewed changes

address comments

6af2eba

maropu reviewed Apr 16, 2020

View reviewed changes

describe window_frame in detail

3fb73f0

maropu reviewed Apr 16, 2020

View reviewed changes

docs/sql-ref-syntax-qry-window.md Outdated

### Examples

{% highlight sql %}

Copy link

Member

maropu Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove this blank.

maropu approved these changes Apr 16, 2020

View reviewed changes

address comments

747cfef

srowen approved these changes Apr 17, 2020

View reviewed changes

maropu reviewed Apr 17, 2020

View reviewed changes

as -> AS

6a3d475

maropu closed this in 142f436 Apr 18, 2020

huaxingao deleted the sql-win-fun branch April 18, 2020 00:36

		### Description

		Window functions operate on a group of rows, referred to as a window, and calculate an aggregated value for each row based on the specified window. Window functions are useful for processing tasks such as calculating a moving average, computing a cumulative, or accessing the value of rows given the relative position of the current row. Spark SQL supports three types of window functions:

[SPARK-31390][SQL][DOCS] Document Window Function in SQL Syntax Section #28220

[SPARK-31390][SQL][DOCS] Document Window Function in SQL Syntax Section #28220

Uh oh!

Conversation

huaxingao commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Apr 15, 2020

Uh oh!

huaxingao commented Apr 15, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropu commented Apr 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 16, 2020

Uh oh!

SparkQA commented Apr 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Apr 15, 2020 •

edited

Loading

maropu left a comment •

edited

Loading