Skip to content

Commit 26eb3f6

Browse files
ScrapCodesScrapCodes
committed
Patrick's comments on PR.
Added a "Upgrading from pre-1.0 versions of Spark" section in the Java programming guide. Added brief README file in the java8-tests directory that explains what it is? Fixed "When running the tests in Maven, all of the output is sent console, and not the test summaries as they were running." Fixed " hard to get SBT to use the correct Java version without setting Java 8 as my system default." added a warning to dev/run-tests script if the Java version is less than 1.8 Moved java8-tests folder into a new folder called /extras
1 parent 35d8d79 commit 26eb3f6

File tree

11 files changed

+100
-91
lines changed

11 files changed

+100
-91
lines changed

‎dev/run-tests

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ rm -rf ./work
2727
# Fail fast
2828
set -e
2929

30+
if test -x "$JAVA_HOME/bin/java"; then
31+
declare java_cmd="$JAVA_HOME/bin/java"
32+
else
33+
declare java_cmd=java
34+
fi
35+
36+
JAVA_VERSION=$($java_cmd -version 2>&1 | sed 's/java version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q')
37+
[ "$JAVA_VERSION" -ge 18 ] && echo "" || echo "[Warn] Java 8 tests will not run, because JDK version is < 1.8."
38+
39+
3040
echo "========================================================================="
3141
echo "Running Scala style checks"
3242
echo "========================================================================="

‎docs/building-with-maven.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,8 @@ The debian package can then be found under assembly/target. We added the short c
8383

8484
Running only java 8 tests and nothing else.
8585

86-
$ mvn test -DskipTests -Pjava8-tests
86+
$ mvn install -DskipTests -Pjava8-tests
8787

88-
Java 8 tests are run when -Pjava8-tests profile is enabled, they will run inspite of -DskipTests. For these tests to run java 8 should be installed on the system running the tests.
88+
Java 8 tests are run when -Pjava8-tests profile is enabled, they will run in spite of -DskipTests.
89+
For these tests to run your system must have a JDK 8 installation.
90+
If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests.

‎docs/java-programming-guide.md

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,21 @@ which support the same methods as their Scala counterparts but take Java functio
1717
Java data and collection types. The main differences have to do with passing functions to RDD
1818
operations (e.g. map) and handling RDDs of different types, as discussed next.
1919

20+
# Upgrading from pre-1.0 versions of Spark
21+
22+
There are following API changes for codebases written in pre-1.0 versions of Spark.
23+
24+
* All `org.apache.spark.api.java.function.*` abstract classes are now interfaces.
25+
So this means that concrete implementations of these `Function` abstract classes will
26+
have `implements` instead of extends.
27+
* APIs of map and flatMap in core and map, flatMap and transform in streaming
28+
are changed and are defined on the basis of the passed anonymous function's
29+
return type, for example mapToPair(...) or flatMapToPair returns
30+
[`JavaPairRDD`](api/core/index.html#org.apache.spark.api.java.JavaPairRDD),
31+
similarly mapToDouble and flatMapToDouble returns
32+
[`JavaDoubleRDD`](api/core/index.html#org.apache.spark.api.java.JavaDoubleRDD).
33+
Please check the API documentation for more details.
34+
2035
# Key Differences in the Java API
2136

2237
There are a few key differences between the Java and Scala APIs:
@@ -30,7 +45,7 @@ There are a few key differences between the Java and Scala APIs:
3045
classes for key-value pairs and doubles. For example,
3146
[`JavaPairRDD`](api/core/index.html#org.apache.spark.api.java.JavaPairRDD)
3247
stores key-value pairs.
33-
* To support upcoming java 8 lambda expression, methods are defined on the basis of
48+
* To support java 8 lambda expression, methods are defined on the basis of
3449
the passed anonymous function's (a.k.a lambda expression) return type,
3550
for example mapToPair(...) or flatMapToPair returns
3651
[`JavaPairRDD`](api/core/index.html#org.apache.spark.api.java.JavaPairRDD),
@@ -139,7 +154,7 @@ lambda expression as follows:
139154
JavaRDD<String> words = lines.flatMap(s -> Arrays.asList(s.split(" ")));
140155
{% endhighlight %}
141156

142-
Same possibility applies to all passed in anonymous classes in java 8.
157+
This lambda syntax can be applied to all anonymous classes in Java 8.
143158

144159
Continuing with the word count example, we map each word to a `(word, 1)` pair:
145160

‎extras/java8-tests/README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Java 8 test suites.
2+
3+
These tests are bundled with spark and run if you have java 8 installed as system default or your `JAVA_HOME` points to a java 8(or higher) installation. `JAVA_HOME` is preferred to system default jdk installation. Since these tests require jdk 8 or higher, they defined to be optional to run in the build system.
4+
5+
For sbt users, it automatically detects the presence of java 8 based on either `JAVA_HOME` environment variable or default installed jdk and run these tests.
6+
7+
For maven users,
8+
9+
This automatic detection is not possible and thus user has to ensure that either `JAVA_HOME` environment variable or default installed jdk points to jdk 8.
10+
11+
`$ mvn install -Pjava8-tests`
12+
13+
Above command can only be run from project root directory since this module depends on both core and test-jars of core and streaming. These jars are installed first time the above command is run as java8-tests profile enables local publishing of test-jar artifacts as well. Once these artifacts are published then these tests can be run from this module's directory as well.

‎java8-tests/pom.xml renamed to ‎extras/java8-tests/pom.xml

Lines changed: 36 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -61,98 +61,55 @@
6161
<profiles>
6262
<profile>
6363
<id>java8-tests</id>
64-
<build>
65-
<plugins>
66-
<plugin>
67-
<groupId>org.apache.maven.plugins</groupId>
68-
<artifactId>maven-surefire-plugin</artifactId>
69-
<executions>
70-
<execution>
71-
<id>test</id>
72-
<goals>
73-
<goal>test</goal>
74-
</goals>
75-
</execution>
76-
</executions>
77-
<configuration>
78-
<skipTests>false</skipTests>
79-
<includes>
80-
<include>**/Suite*.java</include>
81-
<include>**/*Suite.java</include>
82-
</includes>
83-
</configuration>
84-
</plugin>
85-
<plugin>
86-
<groupId>org.apache.maven.plugins</groupId>
87-
<artifactId>maven-compiler-plugin</artifactId>
88-
<executions>
89-
<execution>
90-
<id>test-compile-first</id>
91-
<phase>process-test-resources</phase>
92-
<goals>
93-
<goal>testCompile</goal>
94-
</goals>
95-
</execution>
96-
</executions>
97-
<configuration>
98-
<fork>true</fork>
99-
<verbose>true</verbose>
100-
<forceJavacCompilerUse>true</forceJavacCompilerUse>
101-
<source>1.8</source>
102-
<compilerVersion>1.8</compilerVersion>
103-
<target>1.8</target>
104-
<encoding>UTF-8</encoding>
105-
<maxmem>1024m</maxmem>
106-
</configuration>
107-
</plugin>
108-
<plugin>
109-
<!-- disabled -->
110-
<groupId>net.alchim31.maven</groupId>
111-
<artifactId>scala-maven-plugin</artifactId>
112-
<executions>
113-
<execution>
114-
<phase>none</phase>
115-
</execution>
116-
<execution>
117-
<id>scala-compile-first</id>
118-
<phase>none</phase>
119-
</execution>
120-
<execution>
121-
<id>scala-test-compile-first</id>
122-
<phase>none</phase>
123-
</execution>
124-
<execution>
125-
<id>attach-scaladocs</id>
126-
<phase>none</phase>
127-
</execution>
128-
</executions>
129-
</plugin>
130-
<plugin>
131-
<groupId>org.scalatest</groupId>
132-
<artifactId>scalatest-maven-plugin</artifactId>
133-
<executions>
134-
<execution>
135-
<id>test</id>
136-
<phase>none</phase>
137-
</execution>
138-
</executions>
139-
</plugin>
140-
</plugins>
141-
</build>
14264
</profile>
14365
</profiles>
14466
<build>
14567
<plugins>
68+
<plugin>
69+
<groupId>org.apache.maven.plugins</groupId>
70+
<artifactId>maven-surefire-plugin</artifactId>
71+
<executions>
72+
<execution>
73+
<id>test</id>
74+
<goals>
75+
<goal>test</goal>
76+
</goals>
77+
</execution>
78+
</executions>
79+
<configuration>
80+
<skipTests>false</skipTests>
81+
<includes>
82+
<include>**/Suite*.java</include>
83+
<include>**/*Suite.java</include>
84+
</includes>
85+
<redirectTestOutputToFile>true</redirectTestOutputToFile>
86+
</configuration>
87+
</plugin>
14688
<plugin>
14789
<groupId>org.apache.maven.plugins</groupId>
14890
<artifactId>maven-compiler-plugin</artifactId>
14991
<executions>
15092
<execution>
151-
<phase>none</phase>
93+
<id>test-compile-first</id>
94+
<phase>process-test-resources</phase>
95+
<goals>
96+
<goal>testCompile</goal>
97+
</goals>
15298
</execution>
15399
</executions>
100+
<configuration>
101+
<fork>true</fork>
102+
<verbose>true</verbose>
103+
<forceJavacCompilerUse>true</forceJavacCompilerUse>
104+
<source>1.8</source>
105+
<compilerVersion>1.8</compilerVersion>
106+
<target>1.8</target>
107+
<encoding>UTF-8</encoding>
108+
<maxmem>1024m</maxmem>
109+
</configuration>
154110
</plugin>
155111
<plugin>
112+
<!-- disabled -->
156113
<groupId>net.alchim31.maven</groupId>
157114
<artifactId>scala-maven-plugin</artifactId>
158115
<executions>

‎java8-tests/src/test/java/org/apache/spark/Java8APISuite.java renamed to ‎extras/java8-tests/src/test/java/org/apache/spark/Java8APISuite.java

File renamed without changes.

‎java8-tests/src/test/java/org/apache/spark/streaming/Java8APISuite.java renamed to ‎extras/java8-tests/src/test/java/org/apache/spark/streaming/Java8APISuite.java

File renamed without changes.

‎pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -732,7 +732,7 @@
732732
</build>
733733

734734
<modules>
735-
<module>java8-tests</module>
735+
<module>extras/java8-tests</module>
736736
</modules>
737737

738738
</profile>

‎project/SparkBuild.scala

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,10 @@ object SparkBuild extends Build {
9595
lazy val javaVersion = System.getProperty("java.specification.version")
9696
lazy val isJava8Enabled = javaVersion.toDouble >= "1.8".toDouble
9797
val maybeJava8Tests = if (isJava8Enabled) Seq[ProjectReference](java8Tests) else Seq[ProjectReference]()
98-
lazy val java8Tests = Project("java8-tests", file("java8-tests"), settings = java8TestsSettings) dependsOn(core) dependsOn(streaming % "compile->compile;test->test")
98+
lazy val java8Tests = Project("java8-tests", file("extras/java8-tests"), settings = java8TestsSettings).
99+
dependsOn(core) dependsOn(streaming % "compile->compile;test->test")
100+
101+
val javaHomeEnv = Properties.envOrNone("JAVA_HOME").map(file)
99102

100103
// Conditionally include the yarn sub-project
101104
lazy val yarnAlpha = Project("yarn-alpha", file("yarn/alpha"), settings = yarnAlphaSettings) dependsOn(core)
@@ -140,6 +143,7 @@ object SparkBuild extends Build {
140143
javacOptions := Seq("-target", JAVAC_JVM_VERSION, "-source", JAVAC_JVM_VERSION),
141144
unmanagedJars in Compile <<= baseDirectory map { base => (base / "lib" ** "*.jar").classpath },
142145
retrieveManaged := true,
146+
if (javaHomeEnv.isDefined) javaHome := javaHomeEnv else javaHome <<= javaHome in Compile,
143147
// This is to add convenience of enabling sbt -Dsbt.offline=true for making the build offline.
144148
offline := "true".equalsIgnoreCase(sys.props("sbt.offline")),
145149
retrievePattern := "[type]s/[artifact](-[revision])(-[classifier]).[ext]",

‎sbt/sbt-launch-lib.bash

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,13 @@ declare -a residual_args
1616
declare -a java_args
1717
declare -a scalac_args
1818
declare -a sbt_commands
19-
declare java_cmd=java
19+
20+
if test -x "$JAVA_HOME/bin/java"; then
21+
echo -e "using $JAVA_HOME for launching sbt."
22+
declare java_cmd="$JAVA_HOME/bin/java"
23+
else
24+
declare java_cmd=java
25+
fi
2026

2127
echoerr () {
2228
echo 1>&2 "$@"

0 commit comments

Comments
 (0)