Skip to content

Commit 5ffc73e

Browse files
zsxwingsrowen
authored andcommitted
[SPARK-5074] [CORE] [TESTS] Fix the flakey test 'run shuffle with map stage failure' in DAGSchedulerSuite
Test failure: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=centos/2240/testReport/junit/org.apache.spark.scheduler/DAGSchedulerSuite/run_shuffle_with_map_stage_failure/ This is because many tests share the same `JobListener`. Because after each test, `scheduler` isn't stopped. So actually it's still running. When running the test `run shuffle with map stage failure`, some previous test may trigger [ResubmitFailedStages](https://github.com/apache/spark/blob/ebc25a4ddfe07a67668217cec59893bc3b8cf730/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1120) logic, and report `jobFailed` and override the global `failure` variable. This PR uses `after` to call `scheduler.stop()` for each test. Author: zsxwing <zsxwing@gmail.com> Closes #5903 from zsxwing/SPARK-5074 and squashes the following commits: 1e6f13e [zsxwing] Fix the flakey test 'run shuffle with map stage failure' in DAGSchedulerSuite
1 parent b83091a commit 5ffc73e

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,10 @@ class DAGSchedulerSuite
174174
dagEventProcessLoopTester = new DAGSchedulerEventProcessLoopTester(scheduler)
175175
}
176176

177+
after {
178+
scheduler.stop()
179+
}
180+
177181
override def afterAll() {
178182
super.afterAll()
179183
}
@@ -261,8 +265,9 @@ class DAGSchedulerSuite
261265
override def taskSucceeded(partition: Int, value: Any) = numResults += 1
262266
override def jobFailed(exception: Exception) = throw exception
263267
}
264-
submit(new MyRDD(sc, 0, Nil), Array(), listener = fakeListener)
268+
val jobId = submit(new MyRDD(sc, 0, Nil), Array(), listener = fakeListener)
265269
assert(numResults === 0)
270+
cancel(jobId)
266271
}
267272

268273
test("run trivial job") {

0 commit comments

Comments
 (0)