Skip to content

[SPARK-8498] [SQL] Add regression test for SPARK-8470 #6909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-8498] [SQL] Add regression test for SPARK-8470 #6909

wants to merge 1 commit into from

Conversation

andrewor14
Copy link
Contributor

Summary of the problem in SPARK-8470. When using HiveContext to create a data frame of a user case class, Spark throws scala.reflect.internal.MissingRequirementError when it tries to infer the schema using reflection. This is caused by HiveContext silently overwriting the context class loader containing the user classes.

What this issue is about. This issue adds regression tests for SPARK-8470, which is already fixed in #6891. We closed SPARK-8470 as a duplicate because it is a different manifestation of the same problem in SPARK-8368. Due to the complexity of the reproduction, this requires us to pre-package a special test jar and include it in the Spark project itself.

I tested this with and without the fix in #6891 and verified that it passes only if the fix is present.

@andrewor14 andrewor14 force-pushed the SPARK-8498 branch 2 times, most recently from 4c3cebd to 7ede573 Compare June 19, 2015 21:47
@andrewor14
Copy link
Contributor Author

@marmbrus @yhuai

Add regression test for SPARK-8470
5e9d688 
Due to the complexity of the reproduction, this requires us to
pre-package a special test jar and include it in the Spark project
itself. This test jar is then used in HiveSparkSubmitSuite.
@andrewor14 andrewor14 changed the title [SPARK-8498] Add regression test for SPARK-8470 [SPARK-8498] [SQL] Add regression test for SPARK-8470 Jun 19, 2015
@andrewor14
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 19, 2015

Test build #35327 has finished for PR 6909 at commit 7ede573.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • // class ParentClass(parentField: Int)
    • // class ChildClass(childField: Int) extends ParentClass(1)
    • // If the class type corresponding to current slot has writeObject() defined,
    • // then its not obvious which fields of the class will be serialized as the writeObject()
    • class StreamingKMeansModel(KMeansModel):
    • class StreamingKMeans(object):
    • abstract class GeneratedClass
    • case class Bin(child: Expression)
    • case class Md5(child: Expression)

@SparkQA
Copy link

SparkQA commented Jun 19, 2015

Test build #35328 has finished for PR 6909 at commit 5e9d688.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • // class ParentClass(parentField: Int)
    • // class ChildClass(childField: Int) extends ParentClass(1)
    • // If the class type corresponding to current slot has writeObject() defined,
    • // then its not obvious which fields of the class will be serialized as the writeObject()
    • class StreamingKMeansModel(KMeansModel):
    • class StreamingKMeans(object):
    • abstract class GeneratedClass
    • case class Md5(child: Expression)

@yhuai
Copy link
Contributor

yhuai commented Jun 20, 2015

Merging it to master and branch 1.4.

asfgit pushed a commit that referenced this pull request Jun 20, 2015
@yhuai
[SPARK-8498] [SQL] Add regression test for SPARK-8470
2248ad8 
**Summary of the problem in SPARK-8470.** When using `HiveContext` to create a data frame of a user case class, Spark throws `scala.reflect.internal.MissingRequirementError` when it tries to infer the schema using reflection. This is caused by `HiveContext` silently overwriting the context class loader containing the user classes.

**What this issue is about.** This issue adds regression tests for SPARK-8470, which is already fixed in #6891. We closed SPARK-8470 as a duplicate because it is a different manifestation of the same problem in SPARK-8368. Due to the complexity of the reproduction, this requires us to pre-package a special test jar and include it in the Spark project itself.

I tested this with and without the fix in #6891 and verified that it passes only if the fix is present.

Author: Andrew Or <andrew@databricks.com>

Closes #6909 from andrewor14/SPARK-8498 and squashes the following commits:

5e9d688 [Andrew Or] Add regression test for SPARK-8470

(cherry picked from commit 093c348)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@asfgit asfgit closed this in 093c348 Jun 20, 2015
@andrewor14 andrewor14 deleted the SPARK-8498 branch June 20, 2015 00:40
@SparkQA
Copy link

SparkQA commented Jun 20, 2015

Test build #35336 has finished for PR 6909 at commit 5e9d688.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • // class ParentClass(parentField: Int)
    • // class ChildClass(childField: Int) extends ParentClass(1)
    • // If the class type corresponding to current slot has writeObject() defined,
    • // then its not obvious which fields of the class will be serialized as the writeObject()
    • class StreamingKMeansModel(KMeansModel):
    • class StreamingKMeans(object):
    • abstract class GeneratedClass
    • case class Bin(child: Expression)
    • case class Md5(child: Expression)

andrewor14
andrewor14 reviewed Jun 22, 2015
View reviewed changes
sql/hive/src/test/resources/regression-test-SPARK-8498/Main.scala
import org.apache.spark.sql.hive.HiveContext

/**
* Entry point in test application for SPARK-8498.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is supposed to be SPARK-8489. This is corrected in a future commit.

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 22, 2015
@nemccarthy
[SPARK-8498] [SQL] Add regression test for SPARK-8470
1a63ce7 
**Summary of the problem in SPARK-8470.** When using `HiveContext` to create a data frame of a user case class, Spark throws `scala.reflect.internal.MissingRequirementError` when it tries to infer the schema using reflection. This is caused by `HiveContext` silently overwriting the context class loader containing the user classes.

**What this issue is about.** This issue adds regression tests for SPARK-8470, which is already fixed in apache#6891. We closed SPARK-8470 as a duplicate because it is a different manifestation of the same problem in SPARK-8368. Due to the complexity of the reproduction, this requires us to pre-package a special test jar and include it in the Spark project itself.

I tested this with and without the fix in apache#6891 and verified that it passes only if the fix is present.

Author: Andrew Or <andrew@databricks.com>

Closes apache#6909 from andrewor14/SPARK-8498 and squashes the following commits:

5e9d688 [Andrew Or] Add regression test for SPARK-8470

(cherry picked from commit 093c348)
Signed-off-by: Yin Huai <yhuai@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewor14 @SparkQA @yhuai