Serialization/classloading issues on spark 1.5 and scala 2.11

[PanAeon]

Hi,
in scala 2.11 a new option -Yrepl-class-based was added to the repl. The main motivation was to make code in repl serializable by default and available for use over the wire. More details here -Yrepl-class-based .
This feature allowed spark to significantly simplify their code for repl for scala-2.11. But it broke scala-notebook.

Symptoms:
when anonymous closure is used in RDD transformation in the local mode it hangs when closure captures local variable in the same cell. In the standalone mode any anonymous closure in a RDD transformation results in error like "ClassNotFoundException - class $anonfun$ is not found".

[andypetrella]

ouch... weird that it only happen in the spark-notebook... how did you tried that? I mean how did you pass this argument, using the metadata?

then whatever transformation will be broken?

[PanAeon]

Here's the code for reproduction, in standalone mode (just specifying spark-master), on scala 2.11 && spark-1.5.2 it should fail miserably:

val rdd =  sc.parallelize(Stream.from(1).take(1000).toList,4)
val simpleVariable = Set(1, 2, 3, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19)
val s =   rdd
  .filter(x => simpleVariable(x)) // throws classdefnotfound exception....
  .collect()

At first I thought it was serialization issue, that I mentioned before. But it is much more serious. Note that it is not valid for scala 2.10, and because spark by default is built for 2.10 many users will not notice it yet. But I like to be on the bleeding edge)

[PanAeon]

Here's the pull request with changes to repl in spark 2.11 apache/spark#6903