Skip to content

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
YanTangZhai wants to merge 11 commits into from
Closed

[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null #3538

YanTangZhai wants to merge 11 commits into from

Conversation

@YanTangZhai
Copy link
Contributor

@YanTangZhai YanTangZhai commented Dec 1, 2014

val jsc = new org.apache.spark.api.java.JavaSparkContext(sc)
val jhc = new org.apache.spark.sql.hive.api.java.JavaHiveContext(jsc)
val nrdd = jhc.hql("select null from spark_test.for_test")
println(nrdd.schema)
Then the error is thrown as follows:
scala.MatchError: NullType (of class org.apache.spark.sql.catalyst.types.NullType$)
at org.apache.spark.sql.types.util.DataTypeConversions$.asJavaDataType(DataTypeConversions.scala:43)

@SparkQA
Copy link

SparkQA commented Dec 1, 2014

Test build #23989 has started for PR 3538 at commit 896c7b7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 1, 2014

Test build #23989 has finished for PR 3538 at commit 896c7b7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 1, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23989/
Test PASSed.

marmbrus
marmbrus reviewed Dec 1, 2014
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala Outdated
Copy link
Contributor

@marmbrus marmbrus Dec 1, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not also add a null type to JDataType?

@marmbrus
Copy link
Contributor

marmbrus commented Dec 1, 2014

It would also be good to add a test case for this to JavaSQLSuite

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24020 has started for PR 3538 at commit 4b4bb34.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24021 has started for PR 3538 at commit e052dff.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24020 has finished for PR 3538 at commit 4b4bb34.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class NullType extends DataType

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 2, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24020/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Dec 2, 2014

Test build #24021 has finished for PR 3538 at commit e052dff.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class NullType extends DataType

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 2, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24021/
Test PASSed.

marmbrus
marmbrus reviewed Dec 2, 2014
View reviewed changes
sql/core/src/main/java/org/apache/spark/sql/api/java/DataType.java
Copy link
Contributor

@marmbrus marmbrus Dec 2, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation is off here.

@asfgit asfgit closed this in 1066427 Dec 2, 2014
asfgit pushed a commit that referenced this pull request Dec 2, 2014
@YanTangZhai @marmbrus
[SPARK-4676][SQL] JavaSchemaRDD.schema may throw NullType MatchError …
06129cd 
…if sql has null

val jsc = new org.apache.spark.api.java.JavaSparkContext(sc)
val jhc = new org.apache.spark.sql.hive.api.java.JavaHiveContext(jsc)
val nrdd = jhc.hql("select null from spark_test.for_test")
println(nrdd.schema)
Then the error is thrown as follows:
scala.MatchError: NullType (of class org.apache.spark.sql.catalyst.types.NullType$)
at org.apache.spark.sql.types.util.DataTypeConversions$.asJavaDataType(DataTypeConversions.scala:43)

Author: YanTangZhai <hakeemzhai@tencent.com>
Author: yantangzhai <tyz0303@163.com>
Author: Michael Armbrust <michael@databricks.com>

Closes #3538 from YanTangZhai/MatchNullType and squashes the following commits:

e052dff [yantangzhai] [SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null
4b4bb34 [yantangzhai] [SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null
896c7b7 [yantangzhai] fix NullType MatchError in JavaSchemaRDD when sql has null
6e643f8 [YanTangZhai] Merge pull request #11 from apache/master
e249846 [YanTangZhai] Merge pull request #10 from apache/master
d26d982 [YanTangZhai] Merge pull request #9 from apache/master
76d4027 [YanTangZhai] Merge pull request #8 from apache/master
03b62b0 [YanTangZhai] Merge pull request #7 from apache/master
8a00106 [YanTangZhai] Merge pull request #6 from apache/master
cbcba66 [YanTangZhai] Merge pull request #3 from apache/master
cdef539 [YanTangZhai] Merge pull request #1 from apache/master

(cherry picked from commit 1066427)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@marmbrus
Copy link
Contributor

marmbrus commented Dec 2, 2014

Thanks for fixing this! I've merged to master and 1.2 (I fixed the indentation while merging).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@YanTangZhai @SparkQA @AmplabJenkins @marmbrus