Skip to content

[SPARK-32890][SQL] Pass all sql/hive module UTs in Scala 2.13 #29760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

LuciferYang
Copy link
Contributor

What changes were proposed in this pull request?

This pr fix failed cases in sql hive module in Scala 2.13 as follow:

  • HiveSchemaInferenceSuite (1 FAILED -> PASS)
  • HiveSparkSubmitSuite (1 FAILED-> PASS)
  • StatisticsSuite (1 FAILED-> PASS)
  • HiveDDLSuite (1 FAILED-> PASS)

After this patch all test passed in sql hive module in Scala 2.13.

Why are the changes needed?

We need to support a Scala 2.13 build.

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Scala 2.12: Pass the Jenkins or GitHub Action

  • Scala 2.13: All tests passed.

Do the following:

dev/change-scala-version.sh 2.13
mvn clean install -DskipTests -pl sql/hive -am -Pscala-2.13 -Phive
mvn clean test -pl sql/hive -Pscala-2.13 -Phive

Before

Tests: succeeded 3662, failed 4, canceled 0, ignored 601, pending 0
*** 4 TESTS FAILED ***

After

Tests: succeeded 3666, failed 0, canceled 0, ignored 601, pending 0
All tests passed.

@LuciferYang
Copy link
Contributor Author

LuciferYang commented Sep 15, 2020

cc @srowen , this pr fix all failed test cases in sql/hive module.

In this patch add test-2.13.jar to sql/hive/src/test/resources/regression-test-SPARK-8489 dir, the test-2.13.jar not a copy of test-2.12.jar, I try to generate test-2.13 as #22308 but failed because there is no spark 3.1.0-SNAPSHOT deploy in maven repository, and I generate it as follow:

  1. create a test module in spark source
  2. copy Main.scala and MyCoolClass.scala to src/main/scala dir of test module
  3. create a pom.xml use spark base pom.xml as parent pom and add dependency
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_${scala.binary.version}</artifactId>
      <version>${project.version}</version>
    </dependency>
  1. mvn build test module with -am and rename original-test-2.13-3.1.0-SNAPSHOT.jar to test-2.13.jar

Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK to me

@@ -129,7 +129,8 @@ class HiveSchemaInferenceSuite
// properties out).
assert(!externalCatalog.getTable(DATABASE, TEST_TABLE_NAME).schemaPreservesCase)
val rawTable = client.getTable(DATABASE, TEST_TABLE_NAME)
assert(rawTable.properties.filterKeys(_.startsWith(DATASOURCE_SCHEMA_PREFIX)) == Map.empty)
assert(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just check isEmpty?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address ce5c81b fix this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check isEmpty looks simpler

@srowen
Copy link
Member

srowen commented Sep 15, 2020

Jenkins test this please

@SparkQA
Copy link

SparkQA commented Sep 15, 2020

Test build #128712 has finished for PR 29760 at commit f0d5880.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 15, 2020

Test build #128713 has finished for PR 29760 at commit f0d5880.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 16, 2020

Test build #128732 has finished for PR 29760 at commit ce5c81b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Sep 16, 2020

Merged to master

@srowen srowen closed this in 7fdb571 Sep 16, 2020
@LuciferYang LuciferYang deleted the sql-hive-test branch June 6, 2022 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants