[SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT specific. #30441

LuciferYang · 2020-11-20T06:37:42Z

What changes were proposed in this pull request?

Move "unused-imports" check config to SparkBuild.scala and make it SBT specific.

Why are the changes needed?

Make unused-imports check for SBT specific.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the Jenkins or GitHub Action

HyukjinKwon · 2020-11-20T06:43:43Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -27,6 +27,7 @@ import scala.collection.JavaConverters._
 import scala.collection.Map
 import scala.collection.immutable
 import scala.collection.mutable.HashMap
+import scala.collection.mutable.HashSet


It's for testing right?

Expect this line to trigger compile error

Looks like it's working in Scala 2.12 , but seems -Wconf:cat=unused-imports:e not working in Scala 2.13

-Wunused:imports working in Scala 2.13

HyukjinKwon

Looks good, pending tests.

SparkQA · 2020-11-20T07:06:31Z

Test build #131403 has finished for PR 30441 at commit 4f4cf1a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-20T07:21:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36006/

SparkQA · 2020-11-20T07:37:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36007/

SparkQA · 2020-11-20T07:53:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36006/

SparkQA · 2020-11-20T08:01:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36007/

SparkQA · 2020-11-20T08:05:02Z

Test build #131402 has finished for PR 30441 at commit b9026f2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

LuciferYang · 2020-11-20T08:28:13Z

@HyukjinKwon done ~ waiting tests

LuciferYang · 2020-11-20T08:47:00Z

graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBase.scala

@@ -17,7 +17,6 @@

 package org.apache.spark.graphx.impl

-import scala.language.higherKinds


checked by Scala 2.13 compiler

LuciferYang · 2020-11-20T08:47:06Z

graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBaseOps.scala

@@ -17,7 +17,6 @@

 package org.apache.spark.graphx.impl

-import scala.language.higherKinds


checked by Scala 2.13 compiler

SparkQA · 2020-11-20T08:57:57Z

Test build #131413 has finished for PR 30441 at commit 70e84c6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-20T08:59:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36012/

SparkQA · 2020-11-20T09:20:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36013/

SparkQA · 2020-11-20T09:22:38Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36012/

LuciferYang · 2020-11-20T09:27:25Z

project/SparkBuild.scala

@@ -230,6 +231,8 @@ object SparkBuild extends PomBuild {
          // see `scalac -Wconf:help` for details
          "-Wconf:cat=deprecation:wv,any:e",
          // 2.13-specific warning hits to be muted (as narrowly as possible) and addressed separately
+          // TODO Enable this option when Scala 2.12 is no longer supported.


@HyukjinKwon A little interesting:

Scala 2.13 think this is a unused import but Scala 2.12 compile failed without this import, so comments -Wunused:imports in Scala 2.13 and left a TODO in this pr

Can you file a JIRA under the Scala 2.13 umbrella JIRA? You can make a comment like TODO(SPARK-XXXXX): blah blah.

Address 911acfe fix this

SparkQA · 2020-11-20T09:45:35Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36013/

SparkQA · 2020-11-20T10:36:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36019/

SparkQA · 2020-11-20T10:46:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36020/

SparkQA · 2020-11-20T11:01:11Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36019/

SparkQA · 2020-11-20T11:05:25Z

Test build #131407 has finished for PR 30441 at commit cffdbcd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-20T11:20:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36022/

SparkQA · 2020-11-20T11:23:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36020/

SparkQA · 2020-11-20T11:51:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36022/

SparkQA · 2020-11-20T11:53:29Z

Test build #131414 has finished for PR 30441 at commit 454c525.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-20T11:59:54Z

Test build #131416 has finished for PR 30441 at commit 6fbbd2c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-11-20T12:27:36Z

Merged to master. Thanks for working on this @LuciferYang.

SparkQA · 2020-11-20T12:32:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36027/

SparkQA · 2020-11-20T13:03:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36027/

SparkQA · 2020-11-20T13:24:17Z

Test build #131421 has finished for PR 30441 at commit 0c48e8d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

LuciferYang added 2 commits November 20, 2020 14:22

move unused-import check to sbt only

a92c80d

add a unused import to test

b9026f2

github-actions bot added BUILD CORE labels Nov 20, 2020

LuciferYang changed the title ~~[SPARK-33441][FOLLOWUP] Make unused-imports check for SBT specific.~~ [SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT specific. Nov 20, 2020

HyukjinKwon reviewed Nov 20, 2020

View reviewed changes

HyukjinKwon approved these changes Nov 20, 2020

View reviewed changes

fix

4f4cf1a

LuciferYang added 2 commits November 20, 2020 16:17

change scala 213 conf

3f72197

revert test

cffdbcd

by scala 2.13

70e84c6

github-actions bot added the GRAPHX label Nov 20, 2020

LuciferYang commented Nov 20, 2020

View reviewed changes

LuciferYang added 2 commits November 20, 2020 16:59

revert

454c525

Merge branch 'upmaster' into SPARK-33441-FOLLOWUP

9752f84

comments Scala 2.13 conifig

6fbbd2c

LuciferYang commented Nov 20, 2020

View reviewed changes

add TODO jira number

911acfe

add :

0c48e8d

HyukjinKwon closed this in 2289389 Nov 20, 2020

LuciferYang deleted the SPARK-33441-FOLLOWUP branch June 6, 2022 03:45

		@@ -17,7 +17,6 @@

		package org.apache.spark.graphx.impl

		import scala.language.higherKinds

[SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT specific. #30441

[SPARK-33441][BUILD][FOLLOWUP] Make unused-imports check for SBT specific. #30441

Uh oh!

Conversation

LuciferYang commented Nov 20, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

LuciferYang commented Nov 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

LuciferYang Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

LuciferYang Nov 20, 2020 •

edited

Loading

LuciferYang Nov 20, 2020 •

edited

Loading

LuciferYang Nov 20, 2020 •

edited

Loading