Skip to content

[SPARK-1210] Prevent ContextClassLoader of Actor from becoming ClassLoader of Executo... #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

[SPARK-1210] Prevent ContextClassLoader of Actor from becoming ClassLoader of Executo... #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
43e0feb
Prevent ContextClassLoader of Actor from becoming ClassLoader of Exec…
ueshin Feb 24, 2014
c6c09b6
Add a test to collect objects of class defined in repl.
ueshin Mar 24, 2014
d79e8c0
Change a parent class loader of ExecutorURLClassLoader.
ueshin Mar 26, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions core/src/main/scala/org/apache/spark/executor/Executor.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,10 @@ private[spark] class Executor(
}
}

// Create our ClassLoader and set it on this thread
// Create our ClassLoader
// do this after SparkEnv creation so can access the SecurityManager
private val urlClassLoader = createClassLoader()
private val replClassLoader = addReplClassLoaderIfNeeded(urlClassLoader)
Thread.currentThread.setContextClassLoader(replClassLoader)

// Akka's message frame size. If task result is bigger than this, we use the block manager
// to send the result back.
Expand Down Expand Up @@ -294,7 +293,7 @@ private[spark] class Executor(
* created by the interpreter to the search path
*/
private def createClassLoader(): ExecutorURLClassLoader = {
val loader = this.getClass.getClassLoader
val loader = Thread.currentThread().getContextClassLoader

// For each of the jars in the jarSet, add them to the class loader.
// We assume each of the files has already been fetched.
Expand Down
11 changes: 11 additions & 0 deletions repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,4 +242,15 @@ class ReplSuite extends FunSuite {
assertContains("res4: Array[Int] = Array(0, 0, 0, 0, 0)", output)
}
}

test("collecting objects of class defined in repl") {
val output = runInterpreter("local[2]",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you verify that this test fails when your changes are not included? I tried removing your changes re-compiling and running this test and it seemed to pass. But maybe I just messed something up...

"""
|case class Foo(i: Int)
|val ret = sc.parallelize((1 to 100).map(Foo), 10).collect
""".stripMargin)
assertDoesNotContain("error:", output)
assertDoesNotContain("Exception", output)
assertContains("ret: Array[Foo] = Array(Foo(1),", output)
}
}