[SPARK-4286] Integrate external shuffle service to coarse grained Mesos mode #3861

tnachen · 2014-12-31T20:10:32Z

No description provided.

SparkQA · 2014-12-31T20:12:33Z

Test build #24969 has started for PR 3861 at commit 5c9fd75.

This patch merges cleanly.

SparkQA · 2014-12-31T21:19:45Z

Test build #24969 has finished for PR 3861 at commit 5c9fd75.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)

AmplabJenkins · 2014-12-31T21:19:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24969/
Test FAILed.

SparkQA · 2014-12-31T23:12:30Z

Test build #24976 has started for PR 3861 at commit 392486e.

This patch merges cleanly.

SparkQA · 2015-01-01T00:22:50Z

Test build #24976 has finished for PR 3861 at commit 392486e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)

AmplabJenkins · 2015-01-01T00:22:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24976/
Test FAILed.

tnachen · 2015-01-01T19:54:56Z

I don't think the test failure is related, and the test passes locally too.

ash211 · 2015-01-02T18:32:00Z

core/src/main/scala/org/apache/spark/SparkContext.scala

@@ -998,7 +998,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
   */
  @DeveloperApi
  override def requestExecutors(numAdditionalExecutors: Int): Boolean = {
-    assert(master.contains("yarn") || dynamicAllocationTesting,
+    assert(master.contains("mesos") || master.contains("yarn") || dynamicAllocationTesting,
      "Requesting executors is currently only supported in YARN mode")


Change this message to be "... only supported in YARN or Mesos modes", and the message below

tnachen · 2015-01-03T02:25:01Z

@ash211 Thanks for the review, updated the PR.

SparkQA · 2015-01-03T02:27:34Z

Test build #25002 has started for PR 3861 at commit a8d036c.

This patch merges cleanly.

SparkQA · 2015-01-03T03:54:41Z

Test build #25002 has finished for PR 3861 at commit a8d036c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)

AmplabJenkins · 2015-01-03T03:54:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25002/
Test PASSed.

tnachen · 2015-01-05T22:37:48Z

@aarondav

SparkQA · 2015-01-06T02:02:36Z

Test build #25082 has started for PR 3861 at commit ae1d91a.

This patch merges cleanly.

SparkQA · 2015-01-06T02:07:35Z

Test build #25083 has started for PR 3861 at commit c601c3f.

This patch merges cleanly.

SparkQA · 2015-01-06T03:11:12Z

Test build #25082 has finished for PR 3861 at commit ae1d91a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)
- implicit class StringToAttributeConversionHelper(val sc: StringContext)

AmplabJenkins · 2015-01-06T03:11:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25082/
Test PASSed.

SparkQA · 2015-01-06T03:14:39Z

Test build #25083 has finished for PR 3861 at commit c601c3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)

AmplabJenkins · 2015-01-06T03:14:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25083/
Test PASSed.

SparkQA · 2015-01-06T22:43:07Z

Test build #25119 has started for PR 3861 at commit c954dc7.

This patch merges cleanly.

SparkQA · 2015-01-06T23:50:36Z

Test build #25119 has finished for PR 3861 at commit c954dc7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "cd %s*; %s ./bin/spark-class ".format(basename, prefixEnv)

AmplabJenkins · 2015-01-06T23:50:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25119/
Test PASSed.

andrewor14 · 2015-01-07T19:35:42Z

core/src/main/scala/org/apache/spark/deploy/worker/StandaloneWorkerShuffleService.scala

@@ -54,6 +53,8 @@ class StandaloneWorkerShuffleService(sparkConf: SparkConf, securityManager: Secu
      require(server == null, "Shuffle server already started")
      logInfo(s"Starting shuffle service on port $port with useSasl = $useSasl")
      server = transportContext.createServer(port)
+    } else {
+      logInfo(s"Skip launching shuffle service as it's not enabled.")


This message doesn't seem particularly useful

andrewor14 · 2015-01-07T19:40:51Z

Hi @tnachen thanks for your work. I will do a closer review within the next few days, but first I have a high level question. IIUC, the changes in this PR basically enable the dynamic allocation feature to be used with Mesos coarse-grained mode. However, in Mesos I believe we express the amount of resources an application wants in terms of cores, not the number of executors. Is the concern raised in SPARK-4751 for standalone mode relevant here?

andrewor14 · 2015-02-13T04:47:00Z

core/src/main/scala/org/apache/spark/executor/CoarseGrainedMesosExecutorBackend.scala

+    driver.sendStatusUpdate(TaskStatus.newBuilder()
+      .setState(TaskState.TASK_RUNNING)
+      .setTaskId(taskInfo.getTaskId)
+      .build)


For readability I would prefer that you put the builder in a val first, and then calling driver.sendStatusUpdate on it. Also, please use parentheses for build() here because it's not a getter without side-effects.

andrewor14 · 2015-02-13T05:26:08Z

@tnachen Thanks for working on this. Before I dive deeper into the implementation there's a main open question that I'd like to address. The external shuffle service is intended to live across the executor lifetime and so launched independently of any Spark application. The service enables dynamic allocation of resources because it can continue to serve an executor's shuffle files after the executor has been killed. However, in this patch the service seems to be started inside the executor backend itself and its fate necessarily tied with the application.

If I understand correctly, the Mesos slave is equivalent to the standalone Worker in that it is long running and lives beyond the lifetime of a particular application. If this is the case, the appropriate place to start the shuffle service would be there instead.

Another issue is that this patch in its current state seems to conflate two issues (1) dynamic allocation and (2) external shuffle service. (1) is what you refer to as auto-scaling on the JIRA, and depends on (2) to work. However, since we already check whether shuffle service is enabled in ExecutorAllocationManager, we shouldn't check it again when launching the Mesos executor. More specifically, I don't understand why we launch the executor two different ways depending on whether (2) is enabled. I believe a better solution is to separate these two concerns and launch the executor the same way we already do today.

tnachen · 2015-02-13T07:47:52Z

@andrewor14 Mesos slave doesn't have the concept of Auxilary service as YARN, that launches a service as part of the Slave. It requires a task to be running to launch anything.

Therefore to have a long running service that lives longer than the task lifetime, I went the route to create a custom Mesos executor that launches the Coarse grained spark executor, and continue to run when the spark executor finishes. The custom executor also launches the external shuffle service.

I agree I didn't really like the diverging path, but I felt like the original method is much simpler as it directly launches the coarse grain executor, and don't want to increase the complexity by making it also have to launch this custom executor.

pwendell · 2015-02-18T21:24:41Z

We spoke a bit offline about this, but my feeling was that the best thing here might be to add a way to launch the shuffle service as a standalone application (initially, not one managed by Mesos) so that it can be shared across Spark applications. That would involve writing some simple launching scripts for it in a similar way to existing daemons we launch, and you'd ask users to launch the shuffle service similar to other storage systems like HDFS. That's very simple and would avoid diverging a lot between Mesos and the other modes. And longer term we could actually have a single shared shuffle service that is scheduled by mesos.

tnachen · 2015-02-18T21:29:43Z

Agree and it's currently being worked on. We can close this PR too.

…uler. Based on apache#3861.

Based on apache#3861.

dragos · 2015-03-12T00:13:34Z

I'd like to take a shot at making the external shuffle service a "daemon". Just to make sure I understand the requirements:

the shuffle service is launched manually on cluster machines
it does not care about spark.shuffle.service.enabled, if it's started -- it just starts
it should not depend on Spark or Scala -- this is the part I'd like to get confirmation. If that's the case, it means we can't use helper scripts like spark-submit or spark-class that would simplify loading configuration files

/cc @tnachen

…mon. This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`: - `sbin/start-shuffle-service.sh` - `sbin/stop-shuffle-service.sh` Specific options can be passed in `SPARK_SHUFFLE_OPTS`. This is picking up work from #3861 /cc tnachen Author: Iulian Dragos <jaguarul@gmail.com> Closes #4990 from dragos/feature/external-shuffle-service and squashes the following commits: 6c2b148 [Iulian Dragos] Import order and wrong name fixup. 07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks. 4dc1f91 [Iulian Dragos] Reviewer’s comments: 8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.

…mon. This allows Mesos deployments to use the shuffle service (and implicitly dynamic allocation). It does so by adding a new "main" class and two corresponding scripts in `sbin`: - `sbin/start-shuffle-service.sh` - `sbin/stop-shuffle-service.sh` Specific options can be passed in `SPARK_SHUFFLE_OPTS`. This is picking up work from apache#3861 /cc tnachen Author: Iulian Dragos <jaguarul@gmail.com> Closes apache#4990 from dragos/feature/external-shuffle-service and squashes the following commits: 6c2b148 [Iulian Dragos] Import order and wrong name fixup. 07804ad [Iulian Dragos] Moved ExternalShuffleService to the `deploy` package + other minor tweaks. 4dc1f91 [Iulian Dragos] Reviewer’s comments: 8145429 [Iulian Dragos] Add an external shuffle service that can be run as a daemon.

…os scheduler This is largely based on extracting the dynamic allocation parts from tnachen's #3861. Author: Iulian Dragos <jaguarul@gmail.com> Closes #4984 from dragos/issue/mesos-coarse-dynamicAllocation and squashes the following commits: 39df8cd [Iulian Dragos] Update tests to latest changes in core. 9d2c9fa [Iulian Dragos] Remove adjustment of executorLimitOption in doKillExecutors. 8b00f52 [Iulian Dragos] Latest round of reviews. 0cd00e0 [Iulian Dragos] Add persistent shuffle directory 15c45c1 [Iulian Dragos] Add dynamic allocation to the Spark coarse-grained scheduler.

tnachen changed the title ~~Integrate external shuffle service to coarse grained Mesos mode~~ [SPARK-4286] Integrate external shuffle service to coarse grained Mesos mode Dec 31, 2014

ash211 reviewed Jan 2, 2015
View reviewed changes

tnachen force-pushed the mesos_shuffle branch from 392486e to a8d036c Compare January 3, 2015 02:24

tnachen force-pushed the mesos_shuffle branch from a8d036c to ae1d91a Compare January 6, 2015 02:00

tnachen force-pushed the mesos_shuffle branch from c601c3f to c954dc7 Compare January 6, 2015 22:37

andrewor14 reviewed Jan 7, 2015
View reviewed changes

andrewor14 reviewed Feb 13, 2015
View reviewed changes

tnachen closed this Feb 18, 2015

dragos added a commit to lightbend/spark that referenced this pull request Mar 10, 2015

[SPARK-4286] Add dynamic allocation to the Spark coarse-grained sched…

cf7e84f

…uler. Based on apache#3861.

This was referenced Mar 10, 2015

[SPARK-4286] Add dynamic allocation to the Spark coarse-grained schedule... #4966

Closed

[WIP] [SPARK-4286] Add dynamic allocation to the Spark coarse-grained schedule... lightbend/spark#1

Closed

dragos added a commit to dragos/spark that referenced this pull request Mar 11, 2015

Add dynamic allocation to the Spark coarse-grained scheduler.

6a259fc

Based on apache#3861.

dragos mentioned this pull request Mar 11, 2015

[SPARK-6287][MESOS] Add dynamic allocation to the coarse-grained Mesos scheduler #4984

Closed

dragos mentioned this pull request Mar 12, 2015

[SPARK-4286] Add an external shuffle service that can be run as a daemon. #4990

Closed

[SPARK-4286] Integrate external shuffle service to coarse grained Mesos mode #3861

[SPARK-4286] Integrate external shuffle service to coarse grained Mesos mode #3861

Uh oh!

Conversation

tnachen commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

AmplabJenkins commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

SparkQA commented Jan 1, 2015

Uh oh!

AmplabJenkins commented Jan 1, 2015

Uh oh!

tnachen commented Jan 1, 2015

Uh oh!

ash211 Jan 2, 2015

Choose a reason for hiding this comment

Uh oh!

tnachen commented Jan 3, 2015

Uh oh!

SparkQA commented Jan 3, 2015

Uh oh!

SparkQA commented Jan 3, 2015

Uh oh!

AmplabJenkins commented Jan 3, 2015

Uh oh!

tnachen commented Jan 5, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

AmplabJenkins commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

AmplabJenkins commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

SparkQA commented Jan 6, 2015

Uh oh!

AmplabJenkins commented Jan 6, 2015

Uh oh!

andrewor14 Jan 7, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Jan 7, 2015

Uh oh!

andrewor14 Feb 13, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Feb 13, 2015

Uh oh!

tnachen commented Feb 13, 2015

Uh oh!

pwendell commented Feb 18, 2015

Uh oh!

tnachen commented Feb 18, 2015

Uh oh!

dragos commented Mar 12, 2015

Uh oh!

Uh oh!