-
Notifications
You must be signed in to change notification settings - Fork 28.6k
[SPARK-3365][SQL]Wrong schema generated for List type #4581
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Can one of the admins verify this patch? |
ok to test |
This LGTM, let's wait for Jenkins. Thanks for working on this! |
add to whitelist |
Test build #27411 has started for PR 4581 at commit
|
@tianyi After rethinking about this, now I tend to the solution you proposed originally on JIRA, namely move the |
Test build #27412 has started for PR 4581 at commit
|
Test build #27411 has finished for PR 4581 at commit
|
Test PASSed. |
Test build #27412 has finished for PR 4581 at commit
|
Test PASSed. |
Thanks! Merging to master and branch-1.3. |
This PR fix the issue SPARK-3365. The reason is Spark generated wrong schema for the type `List` in `ScalaReflection.scala` for example: the generated schema for type `Seq[String]` is: ``` {"name":"x","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}` ``` the generated schema for type `List[String]` is: ``` {"name":"x","type":{"type":"struct","fields":[]},"nullable":true,"metadata":{}}` ``` Author: tianyi <tianyi.asiainfo@gmail.com> Closes #4581 from tianyi/SPARK-3365 and squashes the following commits: a097e86 [tianyi] change the order of resolution in ScalaReflection.scala (cherry picked from commit 1c8633f) Signed-off-by: Cheng Lian <lian@databricks.com>
This PR fix the issue SPARK-3365.
The reason is Spark generated wrong schema for the type
List
inScalaReflection.scala
for example:
the generated schema for type
Seq[String]
is:the generated schema for type
List[String]
is: