[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types #30654

yaooqinn · 2020-12-07T18:36:54Z

What changes were proposed in this pull request?

Add migration guide for CHAR VARCHAR types

Why are the changes needed?

for migration

Does this PR introduce any user-facing change?

doc change

How was this patch tested?

passing ci

…AR types

SparkQA · 2020-12-07T18:50:37Z

Test build #132386 has finished for PR 30654 at commit 3cb8724.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-07T19:24:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36987/

SparkQA · 2020-12-07T19:57:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36987/

maropu · 2020-12-08T01:03:28Z

docs/sql-migration-guide.md

@@ -54,6 +54,8 @@ license: |

  - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

+  - In Spark 3.1, CHAR/CHARACTER and VARCHAR types become individual types from string. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, where treats them as string with length parameter simply ignored, you can set `spark.sql.legacy.charVarcharAsString` to `true`.


how about CHAR/CHARACTER and VARCHAR types become individual types from string -> we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types?

nit: table schema -> a table schema

, where treats them as string with length parameter simply ignored, -> , which treats them as STRING types and ignores a length parameter (e.g., CHAR(4)), ?

thanks, updated

…AR types

maropu · 2020-12-08T02:04:04Z

docs/sql-migration-guide.md

@@ -54,6 +54,8 @@ license: |

  - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

+  - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.


Ur, one more nit comment: By default, they can... -> Currently, they can...?

maropu

cc: @cloud

…AR types

SparkQA · 2020-12-08T02:49:29Z

Test build #132396 has finished for PR 30654 at commit 80d4421.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-08T03:15:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36996/

SparkQA · 2020-12-08T03:43:07Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36996/

dongjoon-hyun · 2020-12-08T20:13:42Z

docs/sql-migration-guide.md

@@ -54,6 +54,8 @@ license: |

  - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

+  - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. Currently, they can only be used in a table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.


I'm not sure about the nuance here. I'll leave this to @cloud-fan .

cloud-fan · 2020-12-09T05:13:48Z

docs/sql-migration-guide.md

@@ -54,6 +54,8 @@ license: |

  - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

+  - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. Currently, they can only be used in a table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.


Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore ...

cloud-fan · 2020-12-09T06:44:09Z

thanks, merging to master/3.1! (doc only change, no need to wait for jenkins)

…AR types ### What changes were proposed in this pull request? Add migration guide for CHAR VARCHAR types ### Why are the changes needed? for migration ### Does this PR introduce _any_ user-facing change? doc change ### How was this patch tested? passing ci Closes #30654 from yaooqinn/SPARK-33641-F. Authored-by: Kent Yao <yaooqinn@hotmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit c88edda) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

SparkQA · 2020-12-09T06:55:44Z

Test build #132466 has finished for PR 30654 at commit dc88a0b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-09T07:49:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37068/

SparkQA · 2020-12-09T07:58:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37068/

yaooqinn added 2 commits December 8, 2020 02:35

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCH…

898172b

…AR types

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCH…

3cb8724

…AR types

github-actions bot added the DOCS label Dec 7, 2020

maropu reviewed Dec 8, 2020

View reviewed changes

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCH…

a50e5ec

…AR types

maropu reviewed Dec 8, 2020

View reviewed changes

maropu approved these changes Dec 8, 2020

View reviewed changes

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCH…

80d4421

…AR types

dongjoon-hyun reviewed Dec 8, 2020

View reviewed changes

dongjoon-hyun requested a review from cloud-fan December 8, 2020 20:14

cloud-fan reviewed Dec 9, 2020

View reviewed changes

typo

dc88a0b

cloud-fan closed this in c88edda Dec 9, 2020

		@@ -54,6 +54,8 @@ license: \|

		- In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

		- In Spark 3.1, CHAR/CHARACTER and VARCHAR types become individual types from string. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, where treats them as string with length parameter simply ignored, you can set `spark.sql.legacy.charVarcharAsString` to `true`.

		@@ -54,6 +54,8 @@ license: \|

		- In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`.

		- In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`.

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types #30654

[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types #30654

Uh oh!

Conversation

yaooqinn commented Dec 7, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 7, 2020

Uh oh!

SparkQA commented Dec 7, 2020

Uh oh!

SparkQA commented Dec 7, 2020

Uh oh!

maropu Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

maropu Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

maropu Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

yaooqinn Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

maropu Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

maropu left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 8, 2020

Uh oh!

SparkQA commented Dec 8, 2020

Uh oh!

SparkQA commented Dec 8, 2020

Uh oh!

dongjoon-hyun Dec 8, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan Dec 9, 2020

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Dec 9, 2020

Uh oh!

SparkQA commented Dec 9, 2020

Uh oh!

SparkQA commented Dec 9, 2020

Uh oh!

SparkQA commented Dec 9, 2020

Uh oh!

Uh oh!