[SPARK-26266][BUILD] Update to Scala 2.12.8

[srowen]

What changes were proposed in this pull request?

Update to Scala 2.12.8

How was this patch tested?

Existing tests.

[SparkQA]

Test build #99666 has finished for PR 23218 at commit b667d37.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Hm, looks like genjavadocplugin is published for individual Scala releases and doesn't exist yet for 2.12.8: https://mvnrepository.com/artifact/com.typesafe.genjavadoc/genjavadoc-plugin . I'll look at whether we can leave it at 2.12.7 or whether to expect it will be released soon.

[dongjoon-hyun]

Now, it's released. Let's try again.

• http://central.maven.org/maven2/com/typesafe/genjavadoc/genjavadoc-plugin_2.12.8/0.11/

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

I build this PR with maven, too. The building itself looks good.
Let's see the Jenkins result.

[SparkQA]

Test build #99679 has finished for PR 23218 at commit b667d37.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4451 has finished for PR 23218 at commit b667d37.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #4452 has finished for PR 23218 at commit b667d37.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Hm, one failure was due to a JVM crash, but it fails twice consistent, with sbt just exiting with status 134. No other failures are logged. Not sure what to make of that!

[felixcheung]

change is ok, but maybe the crash is related to .7 -> .8?

[dongjoon-hyun]

Surprisingly, all of three are due to consistent JVM crashes. It seems that Scala 2.12.8, scalatest, or Spark has some unstable code somewhere.

• https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99679/consoleFull

[info] <span style="color: #00CD00;">- SPARK-17641: collect functions should not collect null values (231 milliseconds)</span>
10:51:04.251 WARN org.apache.spark.sql.execution.window.WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
10:51:04.262 WARN org.apache.spark.sql.execution.window.WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa843744e44, pid=116353, tid=140360030242560

• https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4451/consoleFull

[info] <span style="color: #00CD00;">- read from textfile (508 milliseconds)</span>
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f60ec641e44, pid=40380, tid=140053491689216
#

• https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4452/consoleFull

[info] <span style="color: #00CD00;">- SPARK-21996 read from text files generated by file sink -- file name has space (532 milliseconds)</span>
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f399e84ee44, pid=106264, tid=139883238606592
#

[dongjoon-hyun]

This might be https://bugs.openjdk.java.net/browse/JDK-8136447 (which is resolved later by other issue)

$ grep build_loop_late_post consoleFull*
consoleFull.4451:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.4451:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.4452:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.4452:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.4452:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.99679:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.99679:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144
consoleFull.99679:# V  [libjvm.so+0x812e44]  PhaseIdealLoop::build_loop_late_post(Node*)+0x144

Our jenkins are using the following affected version.

JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)

cc @shaneknapp

[srowen]

Ah OK, so all of them were a JVM crash. It would probably be a good idea to update the JVM on all the workers as _60 is over 3 years old. It's probably not as simple as it sounds but WDYT @shaneknapp ?

[shaneknapp]

shouldn't be too hard, but it will require some downtime.

…

On Wed, Dec 5, 2018 at 5:41 AM Sean Owen ***@***.***> wrote: Ah OK, so all of them were a JVM crash. It would probably be a good idea to update the JVM on all the workers as _60 is over 3 years old. It's probably not as simple as it sounds but WDYT @shaneknapp <https://github.com/shaneknapp> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23218 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrB6g0RYWoCxPtgI1FXp-xjCqobIUks5u180QgaJpZM4ZAs01> .

[shaneknapp]

https://issues.apache.org/jira/browse/SPARK-26282

[dongjoon-hyun]

Thank you, @srowen , @HyukjinKwon , @shaneknapp !

[felixcheung]

do we need to relnote jvm compatibility?

[srowen]

I'm not sure. I can't find any other reference to this crash and 2.12.8. It could be something only Spark happens to trigger, or could be specific to this JVM + platform but not Spark or Scala. We could drop a release note at least recommending the latest version of Java 8 (and 11) with Spark 2.4 / 3.0

[shaneknapp]

okie dokie, java8 update is done!
https://issues.apache.org/jira/browse/SPARK-26282

[dongjoon-hyun]

@shaneknapp .
The above run is still using old one. Do we need to retrigger to use JDK8_191?

[info] Building Spark (w/Hive 1.2.1) using SBT with these arguments:  -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly
Using /usr/java/jdk1.8.0_60 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.

[shaneknapp]

oh crap forgot about the PRB. updating the config now.

…

On Fri, Dec 7, 2018 at 2:58 PM Dongjoon Hyun ***@***.***> wrote: @shaneknapp <https://github.com/shaneknapp> . The above run is still using old one. Do we need to retrigger to use JDK8_191? [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly Using /usr/java/jdk1.8.0_60 as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23218 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABiDrGWEM95ltU2VLf_5LITLbplG-y5oks5u2vKigaJpZM4ZAs01> .

[shaneknapp]

ok, PRB builds updated w/the new JAVA_HOME

[dongjoon-hyun]

Retest this please.

[shaneknapp]

it's building w/the proper java now. from the build log:

[EnvInject] - Injecting as environment variables the properties content 
JENKINS_MASTER_HOSTNAME=amp-jenkins-master
JAVA_HOME=/usr/java/jdk1.8.0_191
JAVA_7_HOME=/usr/java/jdk1.7.0_79
SPARK_TESTING=1

[SparkQA]

Test build #4461 has finished for PR 23218 at commit b667d37.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #99849 has finished for PR 23218 at commit b667d37.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master. I'll open a separate PR for branch-2.4