[SPARK-30962][SQL][DOC]Documentation for Alter table command phase 2 #27779

kevinyu98 · 2020-03-03T22:15:14Z

What changes were proposed in this pull request?

Why are the changes needed?

Based on JIRA 30962, we want to add all the support Alter Table syntax for V1 table.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Before:
The documentation looks like
Alter Table

After:

Notes:
Those syntaxes are not supported by v1 Table.

ALTER TABLE .. RENAME COLUMN
ALTER TABLE ... DROP (COLUMN | COLUMNS)
ALTER TABLE ... (ALTER | CHANGE) COLUMN? alterColumnAction only support change comments, not other actions: datatype, position, (SET | DROP) NOT NULL
ALTER TABLE .. CHANGE COLUMN?
ALTER TABLE .... REPLACE COLUMNS
ALTER TABLE ... RECOVER PARTITIONS

kevinyu98 · 2020-03-03T22:15:34Z

@huaxingao @dilipbiswal

HyukjinKwon · 2020-03-04T02:08:30Z

ok to test

SparkQA · 2020-03-04T02:25:35Z

Test build #119271 has finished for PR 27779 at commit 6f74d6c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-03-05T00:07:29Z

docs/sql-ref-syntax-ddl-alter-table.md

Nit: columns -> column?
I guess you missed or after to be altered

Thanks, changed.

huaxingao · 2020-03-05T00:08:33Z

docs/sql-ref-syntax-ddl-alter-table.md

I saw the following in SqlBase.g4. Can alterColumnAction change type, colPosition, setOrDrop as well?

alterColumnAction : TYPE dataType | commentSpec | colPosition | setOrDrop=(SET | DROP) NOT NULL ;

I tested those: datatype, colPosition, set/drop not null, I got error which states v1 table can't do or only supported with v2 tables. So I didn't put those in the doc.

huaxingao · 2020-03-05T00:08:44Z

docs/sql-ref-syntax-ddl-alter-table.md

Does RECOVER PARTITIONS work for V1 as well?

No, it didn't recovery the partition.

nchammas · 2020-03-07T01:21:47Z

docs/sql-ref-syntax-ddl-alter-table.md

It's possible to add multiple partitions in one go, so we should show that in the syntax and examples.

For reference, here's how I wrote up the syntax for ADD PARTITION: https://github.com/apache/spark/pull/27840/files#diff-1029db085c19beee649492b52c628596R93-R95

Yes, it is possible, I will add. Thanks

maropu · 2020-03-08T09:01:17Z

docs/sql-ref-syntax-ddl-alter-table.md

COLUMN is optional?

changed, thanks.

maropu · 2020-03-08T09:03:33Z

docs/sql-ref-syntax-ddl-alter-table.md

nit: new partition -> a new partition

maropu · 2020-03-08T09:03:40Z

docs/sql-ref-syntax-ddl-alter-table.md

nit: new partition -> a new partition

SparkQA · 2020-03-09T19:24:11Z

Test build #119583 has finished for PR 27779 at commit 1d7aa01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-03-09T23:54:00Z

docs/sql-ref-syntax-ddl-alter-table.md

+{% highlight sql %}
+ALTER TABLE table_identifier ADD [IF NOT EXISTS] 
+  PARTITION <partition_spec> 
+  [ PARTITION <partition_spec> ...]


change to

ALTER TABLE table_identifier ADD [ IF NOT EXISTS ] ( partition_spec [ , partition_spec ... ] )

so the format is consistent with others?

, is not part of the syntax in Spark.

I will make changes. Thanks

@nchammas You are right. No ,. Thanks for catching this.

huaxingao · 2020-03-09T23:54:20Z

docs/sql-ref-syntax-ddl-alter-table.md

 ### Related Statements
 - [CREATE TABLE](sql-ref-syntax-ddl-create-table.html)
 - [DROP TABLE](sql-ref-syntax-ddl-drop-table.html)
+- [ALTER TABLE ... ADD PARTITION](sql-ref-syntax-ddl-alter-table.html)


is this change intentional?

oh, I thought that one is part of umbrella jira, so tried to reference. Thanks for catching this, I will remove it.

SparkQA · 2020-03-10T02:32:42Z

Test build #119595 has finished for PR 27779 at commit 6c19a94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen

Text looks OK to me.

huaxingao · 2020-03-10T17:33:39Z

docs/sql-ref-syntax-ddl-alter-table.md

+
+#### Syntax
+{% highlight sql %}
+ALTER TABLE table_identifier { ALTER | CHANGE } [COLUMN] col_spec alterColumnAction


super nit: [COLUMN] -> [ COLUMN ]

You forgot to address the @huajianmao comment?

maropu · 2020-03-10T23:00:57Z

Pending the fix: #27779 (comment)

### What changes were proposed in this pull request? ### Why are the changes needed? Based on [JIRA 30962](https://issues.apache.org/jira/browse/SPARK-30962), we want to add all the support `Alter Table` syntax for V1 table. ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Before: The documentation looks like [Alter Table](#25590) After: <img width="850" alt="Screen Shot 2020-03-03 at 2 02 23 PM" src="https://user-images.githubusercontent.com/7550280/75824837-168c7e00-5d59-11ea-9751-d1dab0f5a892.png"> <img width="977" alt="Screen Shot 2020-03-03 at 2 02 41 PM" src="https://user-images.githubusercontent.com/7550280/75824859-21dfa980-5d59-11ea-8b49-3adf6eb55fc6.png"> <img width="1028" alt="Screen Shot 2020-03-03 at 2 02 59 PM" src="https://user-images.githubusercontent.com/7550280/75824884-2e640200-5d59-11ea-81ef-d77d0a8efee2.png"> <img width="864" alt="Screen Shot 2020-03-03 at 2 03 14 PM" src="https://user-images.githubusercontent.com/7550280/75824910-39b72d80-5d59-11ea-84d0-bffa2499f086.png"> <img width="823" alt="Screen Shot 2020-03-03 at 2 03 28 PM" src="https://user-images.githubusercontent.com/7550280/75824937-45a2ef80-5d59-11ea-932c-314924856834.png"> <img width="811" alt="Screen Shot 2020-03-03 at 2 03 42 PM" src="https://user-images.githubusercontent.com/7550280/75824965-4cc9fd80-5d59-11ea-815b-8c1ebad310b1.png"> <img width="827" alt="Screen Shot 2020-03-03 at 2 03 53 PM" src="https://user-images.githubusercontent.com/7550280/75824978-518eb180-5d59-11ea-8a55-2fa26376b9c1.png"> <img width="783" alt="Screen Shot 2020-03-03 at 2 04 03 PM" src="https://user-images.githubusercontent.com/7550280/75825001-5bb0b000-5d59-11ea-8dd9-dcfbfa1b4330.png"> Notes: Those syntaxes are not supported by v1 Table. - `ALTER TABLE .. RENAME COLUMN` - `ALTER TABLE ... DROP (COLUMN | COLUMNS)` - `ALTER TABLE ... (ALTER | CHANGE) COLUMN? alterColumnAction` only support change comments, not other actions: `datatype, position, (SET | DROP) NOT NULL` - `ALTER TABLE .. CHANGE COLUMN?` - `ALTER TABLE .... REPLACE COLUMNS` - `ALTER TABLE ... RECOVER PARTITIONS` - Closes #27779 from kevinyu98/spark-30962-alterT. Authored-by: Qianyang Yu <qyu@us.ibm.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org> (cherry picked from commit 0f54dc7) Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

maropu · 2020-03-10T23:48:20Z

Thanks! Merged to master/branch-3.0. cc: @gatorsmile

### What changes were proposed in this pull request? ### Why are the changes needed? Based on [JIRA 30962](https://issues.apache.org/jira/browse/SPARK-30962), we want to add all the support `Alter Table` syntax for V1 table. ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Before: The documentation looks like [Alter Table](apache#25590) After: <img width="850" alt="Screen Shot 2020-03-03 at 2 02 23 PM" src="https://user-images.githubusercontent.com/7550280/75824837-168c7e00-5d59-11ea-9751-d1dab0f5a892.png"> <img width="977" alt="Screen Shot 2020-03-03 at 2 02 41 PM" src="https://user-images.githubusercontent.com/7550280/75824859-21dfa980-5d59-11ea-8b49-3adf6eb55fc6.png"> <img width="1028" alt="Screen Shot 2020-03-03 at 2 02 59 PM" src="https://user-images.githubusercontent.com/7550280/75824884-2e640200-5d59-11ea-81ef-d77d0a8efee2.png"> <img width="864" alt="Screen Shot 2020-03-03 at 2 03 14 PM" src="https://user-images.githubusercontent.com/7550280/75824910-39b72d80-5d59-11ea-84d0-bffa2499f086.png"> <img width="823" alt="Screen Shot 2020-03-03 at 2 03 28 PM" src="https://user-images.githubusercontent.com/7550280/75824937-45a2ef80-5d59-11ea-932c-314924856834.png"> <img width="811" alt="Screen Shot 2020-03-03 at 2 03 42 PM" src="https://user-images.githubusercontent.com/7550280/75824965-4cc9fd80-5d59-11ea-815b-8c1ebad310b1.png"> <img width="827" alt="Screen Shot 2020-03-03 at 2 03 53 PM" src="https://user-images.githubusercontent.com/7550280/75824978-518eb180-5d59-11ea-8a55-2fa26376b9c1.png"> <img width="783" alt="Screen Shot 2020-03-03 at 2 04 03 PM" src="https://user-images.githubusercontent.com/7550280/75825001-5bb0b000-5d59-11ea-8dd9-dcfbfa1b4330.png"> Notes: Those syntaxes are not supported by v1 Table. - `ALTER TABLE .. RENAME COLUMN` - `ALTER TABLE ... DROP (COLUMN | COLUMNS)` - `ALTER TABLE ... (ALTER | CHANGE) COLUMN? alterColumnAction` only support change comments, not other actions: `datatype, position, (SET | DROP) NOT NULL` - `ALTER TABLE .. CHANGE COLUMN?` - `ALTER TABLE .... REPLACE COLUMNS` - `ALTER TABLE ... RECOVER PARTITIONS` - Closes apache#27779 from kevinyu98/spark-30962-alterT. Authored-by: Qianyang Yu <qyu@us.ibm.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

huaxingao reviewed Mar 5, 2020

View reviewed changes

huaxingao mentioned this pull request Mar 7, 2020

[SPARK-31075] Add documentation for ALTER TABLE ... ADD PARTITION #27840

Closed

nchammas reviewed Mar 7, 2020

View reviewed changes

maropu changed the title ~~[SPARK-30962][DOC]Documentation for Alter table command phase 2~~ [SPARK-30962][SQL][DOC]Documentation for Alter table command phase 2 Mar 8, 2020

maropu reviewed Mar 8, 2020

View reviewed changes

kevinyu98 added 2 commits March 8, 2020 21:36

alter table phase 2

f945ce1

address comments

1d7aa01

maropu approved these changes Mar 9, 2020

View reviewed changes

kevinyu98 force-pushed the spark-30962-alterT branch from 6f74d6c to 1d7aa01 Compare March 9, 2020 18:57

huaxingao reviewed Mar 9, 2020

View reviewed changes

address comments

6c19a94

srowen reviewed Mar 10, 2020

View reviewed changes

huaxingao reviewed Mar 10, 2020

View reviewed changes

add space

64d131c

maropu closed this in 0f54dc7 Mar 10, 2020

[SPARK-30962][SQL][DOC]Documentation for Alter table command phase 2 #27779

[SPARK-30962][SQL][DOC]Documentation for Alter table command phase 2 #27779

Uh oh!

Conversation

kevinyu98 commented Mar 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

kevinyu98 commented Mar 3, 2020

Uh oh!

HyukjinKwon commented Mar 4, 2020

Uh oh!

SparkQA commented Mar 4, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 10, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu commented Mar 10, 2020

Uh oh!

maropu commented Mar 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

kevinyu98 commented Mar 3, 2020 •

edited

Loading