Skip to content

Commit ac54156

Browse files
committed
[SPARK-30534][INFRA] Use mvn in dev/scalastyle
This PR aims to use `mvn` instead of `sbt` in `dev/scalastyle` to recover GitHub Action. As of now, Apache Spark sbt build is broken by the Maven Central repository policy. https://stackoverflow.com/questions/59764749/requests-to-http-repo1-maven-org-maven2-return-a-501-https-required-status-an > Effective January 15, 2020, The Central Maven Repository no longer supports insecure > communication over plain HTTP and requires that all requests to the repository are > encrypted over HTTPS. We can reproduce this locally by the following. ``` $ rm -rf ~/.m2/repository/org/apache/apache/18/ $ build/sbt clean ``` And, in GitHub Action, `lint-scala` is the only one which is using `sbt`. No. First of all, GitHub Action should be recovered. Also, manually, do the following. **Without Scalastyle violation** ``` $ dev/scalastyle OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0 Using `mvn` from path: /usr/local/bin/mvn Scalastyle checks passed. ``` **With Scalastyle violation** ``` $ dev/scalastyle OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0 Using `mvn` from path: /usr/local/bin/mvn Scalastyle checks failed at following occurrences: error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/main/scala/org/apache/spark/SparkConf.scala message=There should be no empty line separating imports in the same group. line=22 column=0 error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala message=There should be no empty line separating imports in the same group. line=22 column=0 ``` Closes #27242 from dongjoon-hyun/SPARK-30534. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 3848999) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 60a908e commit ac54156

File tree

3 files changed

+6
-17
lines changed

3 files changed

+6
-17
lines changed

dev/scalastyle

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -17,21 +17,10 @@
1717
# limitations under the License.
1818
#
1919

20-
# NOTE: echo "q" is needed because SBT prompts the user for input on encountering a build file
21-
# with failure (either resolution or compilation); the "q" makes SBT quit.
22-
ERRORS=$(echo -e "q\n" \
23-
| build/sbt \
24-
-Pkinesis-asl \
25-
-Pmesos \
26-
-Pkafka-0-8 \
27-
-Pkubernetes \
28-
-Pyarn \
29-
-Pflume \
30-
-Phive \
31-
-Phive-thriftserver \
32-
scalastyle test:scalastyle \
33-
| awk '{if($1~/error/)print}' \
34-
)
20+
SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"
21+
22+
SPARK_PROFILES=${1:-"-Pmesos -Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pdocker-integration-tests -Pkubernetes-integration-tests"}
23+
ERRORS=$($SCRIPT_DIR/../build/mvn $SPARK_PROFILES scalastyle:check | grep "^error file")
3524

3625
if test ! -z "$ERRORS"; then
3726
echo -e "Scalastyle checks failed at following occurrences:\n$ERRORS"

external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with SharedSQLCo
154154
// A value with fractions from DECIMAL(3, 2) is correct:
155155
assert(row.getDecimal(1).compareTo(BigDecimal.valueOf(1.23)) == 0)
156156
// A value > Int.MaxValue from DECIMAL(10) is correct:
157-
assert(row.getDecimal(2).compareTo(BigDecimal.valueOf(9999999999l)) == 0)
157+
assert(row.getDecimal(2).compareTo(BigDecimal.valueOf(9999999999L)) == 0)
158158
}
159159

160160

pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2542,7 +2542,7 @@
25422542
<configuration>
25432543
<verbose>false</verbose>
25442544
<failOnViolation>true</failOnViolation>
2545-
<includeTestSourceDirectory>false</includeTestSourceDirectory>
2545+
<includeTestSourceDirectory>true</includeTestSourceDirectory>
25462546
<failOnWarning>false</failOnWarning>
25472547
<sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
25482548
<testSourceDirectory>${basedir}/src/test/scala</testSourceDirectory>

0 commit comments

Comments
 (0)