[SPARK-31682][SQL] Turn on spark.sql.legacy.createHiveTableByDefault.enabled by default #28500
Conversation
| @@ -2235,7 +2235,7 @@ object SQLConf { | |||
| s"instead of the value of ${DEFAULT_DATA_SOURCE_NAME.key}.") | |||
| .version("3.0.0") | |||
| .booleanConf | |||
| .createWithDefault(false) | |||
| .createWithDefault(true) | |||
There was a problem hiding this comment.
Also check the migration guide. We need to remove one line/
|
Test build #122507 has finished for PR 28500 at commit
|
|
Test build #122509 has finished for PR 28500 at commit
|
|
Hi, @Ngone51 . |
|
+1 for this in master as an intermediate step. -1 for this in 3.0 because it is not safe to release 3.0 with SPARK-30098, even with the property turned off. That PR introduces confusing behavior for CREATE TABLE that is not well understood. I don't think it is a good idea to include SPARK-30098 because we would have to support the flag with that behavior. |
There was a problem hiding this comment.
Hi, @Ngone51 . Since you are already an experienced contributor, I want to add some comments.
First of all, this is not SPARK-30098.
[SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
Also, technically, I tried this as SPARK-31136 officially. And, at that time, @cloud-fan and @HyukjinKwon were there. Since SPARK-31136 was not accepted and the JIRA issue is closed. I only made some test cases neutral to the conf.
Given the updated circumstance (the discussions and email), I agree with this. However, we had better file a new JIRA as a subtask of SPARK-31085 and update this PR's title. This decision is worth of it. Thanks.
|
The discussion thread is still active. Let's make a consensus there before moving on. |
Ngone51
changed the title
Thanks for your advice. Updated. |
|
Thank you so much, @Ngone51 . |
|
Since #28517 is merged, I'll close this PR. Thank you all. |
What changes were proposed in this pull request?
Set
spark.sql.legacy.createHiveTableByDefault.enabledtotrueby default.Why are the changes needed?
According to the latest status of [DISCUSS] Resolve ambiguous parser rule between two "create table"s, we decide to turn this conf on by default to unblock Spark 3.0 release.
Does this PR introduce any user-facing change?
No(comparing to Spark 2.4)
How was this patch tested?
Pass jenkins.