Skip to content

[SPARK-3365][SQL]Wrong schema generated for List type #4581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

tianyi
Copy link
Contributor

@tianyi tianyi commented Feb 13, 2015

This PR fix the issue SPARK-3365.
The reason is Spark generated wrong schema for the type List in ScalaReflection.scala
for example:

the generated schema for type Seq[String] is:

{"name":"x","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}`

the generated schema for type List[String] is:

{"name":"x","type":{"type":"struct","fields":[]},"nullable":true,"metadata":{}}`

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@liancheng
Copy link
Contributor

ok to test

@liancheng
Copy link
Contributor

This LGTM, let's wait for Jenkins. Thanks for working on this!

@liancheng
Copy link
Contributor

add to whitelist

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27411 has started for PR 4581 at commit a8bd6fe.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor

@tianyi After rethinking about this, now I tend to the solution you proposed originally on JIRA, namely move the Product clause below Seq. Usually types like List[T] which extend Seq[T] with Product should be viewed as a Seq at the first place.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27412 has started for PR 4581 at commit a097e86.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27411 has finished for PR 4581 at commit a8bd6fe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27411/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27412 has finished for PR 4581 at commit a097e86.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27412/
Test PASSed.

@liancheng
Copy link
Contributor

Thanks! Merging to master and branch-1.3.

asfgit pushed a commit that referenced this pull request Feb 13, 2015
This PR fix the issue SPARK-3365.
The reason is Spark generated wrong schema for the type `List` in `ScalaReflection.scala`
for example:

the generated schema for type `Seq[String]` is:
```
{"name":"x","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}`
```

the generated schema for type `List[String]` is:
```
{"name":"x","type":{"type":"struct","fields":[]},"nullable":true,"metadata":{}}`
```

Author: tianyi <tianyi.asiainfo@gmail.com>

Closes #4581 from tianyi/SPARK-3365 and squashes the following commits:

a097e86 [tianyi] change the order of resolution in ScalaReflection.scala

(cherry picked from commit 1c8633f)
Signed-off-by: Cheng Lian <lian@databricks.com>
@asfgit asfgit closed this in 1c8633f Feb 13, 2015
@tianyi tianyi deleted the SPARK-3365 branch February 14, 2015 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants