Skip to content

[SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-8489 test for Scala 2.12 #22308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-8489 test for Scala 2.12 #22308

wants to merge 1 commit into from

Conversation

da-liii
Copy link
Contributor

@da-liii da-liii commented Sep 1, 2018

What changes were proposed in this pull request?

remove test-2.10.jar and add test-2.12.jar.

How was this patch tested?

$ sbt -Dscala-2.12
> ++ 2.12.6
> project hive
> testOnly *HiveSparkSubmitSuite -- -z "8489"

@SparkQA
Copy link

SparkQA commented Sep 1, 2018

Test build #95576 has finished for PR 22308 at commit 221fcac.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Sep 1, 2018

Oh what's the change here in the .jar file? just copied the jar file for 2.11/2.10?

@da-liii
Copy link
Contributor Author

da-liii commented Sep 2, 2018

Remove the deprecated 2.10 jar.

And build a 2.12 jar.

@da-liii
Copy link
Contributor Author

da-liii commented Sep 2, 2018

see #12924 and #11744

@da-liii
Copy link
Contributor Author

da-liii commented Sep 2, 2018

@cloud-fan @srowen

This PR will make our jenkins build green.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/204/testReport/

felixcheung
felixcheung reviewed Sep 2, 2018
View reviewed changes
Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we be careful about having jar in the source tree? (that goes into a release?)

@srowen
Copy link
Member

srowen commented Sep 2, 2018

@sadhen I mean is this just a copy of the 2.10 jar? what's in the jar?
@felixcheung in this case there's a test specifically testing loading of a jar file, so it has to have a jar file to load. This code isn't part of the non-test code. (The source is there too FWIW)

@da-liii
Copy link
Contributor Author

da-liii commented Sep 3, 2018

@srowen The 2.12 jar is compiled and packaged from Main.scala and MyCoolClass.scala. Not a copy of 2.10 jar. Diff it, you will verify it.

The steps to generate it:

mvn install // install the 2.4.0-SNAPSHOT at project root
sbt package // package at sql/hive/src/test/resources/regression-test-SPARK-8489

This is the build.sbt:

scalaVersion := 2.12.6
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % "2.4.0-SNAPSHOT"
)

srowen
srowen approved these changes Sep 3, 2018
View reviewed changes
Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that's fine. That makes sense, and it's what the new JAR should be.

@srowen
Copy link
Member

srowen commented Sep 3, 2018

Merged to master

@asfgit asfgit closed this in 64bbd13 Sep 3, 2018
@LuciferYang LuciferYang mentioned this pull request Sep 15, 2020
Closed
@LuciferYang LuciferYang mentioned this pull request Sep 25, 2023
Closed
LuciferYang added a commit that referenced this pull request Sep 26, 2023
@LuciferYang
[SPARK-45321][TESTS] Clean up the unnecessary Scala 2.12 related bina…
2094e71 
…ry files

### What changes were proposed in this pull request?
The purpose of this pr is to clean up the binary files used to assist with Scala 2.12 testing.

They include:
- `core/src/test/resources/TestHelloV3_2.12.jar` and `core/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44246(#41789).
- `connector/connect/client/jvm/src/test/resources/udf2.12` and `connector/connect/client/jvm/src/test/resources/udf2.12.jar` added by SPARK-43744(#42069)
- `connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44293(#41844)
- `sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar` added by SPARK-25304(#22308)

### Why are the changes needed?
Spark 4.0 no longer supports Scala 2.12.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #43106 from LuciferYang/SPARK-45321.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Aug 7, 2024
@LuciferYang @dongjoon-hyun
[SPARK-45321][TESTS] Clean up the unnecessary Scala 2.12 related bina…
c94668d 
…ry files

### What changes were proposed in this pull request?
The purpose of this pr is to clean up the binary files used to assist with Scala 2.12 testing.

They include:
- `core/src/test/resources/TestHelloV3_2.12.jar` and `core/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44246(apache#41789).
- `connector/connect/client/jvm/src/test/resources/udf2.12` and `connector/connect/client/jvm/src/test/resources/udf2.12.jar` added by SPARK-43744(apache#42069)
- `connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar` added by SPARK-44293(apache#41844)
- `sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar` added by SPARK-25304(apache#22308)

### Why are the changes needed?
Spark 4.0 no longer supports Scala 2.12.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#43106 from LuciferYang/SPARK-45321.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@felixcheung felixcheung felixcheung left review comments

@srowen srowen srowen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@da-liii @SparkQA @srowen @felixcheung