SPARK-2641: Passing num executors to spark arguments from properties file by kjsingh · Pull Request #1657 · apache/spark

kjsingh · 2014-07-30T09:40:14Z

Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances.

…e for num executors

AmplabJenkins · 2014-07-30T09:42:25Z

Can one of the admins verify this patch?

pwendell · 2014-08-01T18:40:19Z

I think currently spark.executor.instances is only used internally. However, it does seem useful here to expose this as a proper SparkConf value. @sryza what do you think on this?

sryza · 2014-08-01T19:03:45Z

This makes sense to me. However, we should also document it and mention that it only currently works for YARN.

andrewor14 · 2014-09-03T23:34:26Z

ok to test

SparkQA · 2014-09-03T23:39:32Z

QA tests have started for PR 1657 at commit d8a5a12.

This patch merges cleanly.

SparkQA · 2014-09-04T01:06:43Z

QA tests have finished for PR 1657 at commit d8a5a12.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-05T23:44:24Z

Can one of the admins verify this patch?

andrewor14 · 2014-09-06T00:50:08Z

ok to test

SparkQA · 2014-09-06T01:43:21Z

QA tests have started for PR 1657 at commit d8a5a12.

This patch merges cleanly.

SparkQA · 2014-09-06T03:03:11Z

QA tests have finished for PR 1657 at commit d8a5a12.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2014-09-09T21:05:38Z

nit: could you make the pr title refer to what's actually changing? The current title sounds very generic, while the change is specific to a single config option.

andrewor14 · 2014-09-09T21:52:51Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

Do we want to do this for yarn only?

In standalone or local mode, the no. of executors is not configurable per job.

andrewor14 · 2014-12-20T03:25:22Z

Ok I'm merging this into master 1.2, 1.1 and 1.0 thanks

…file Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances. Author: Kanwaljit Singh <kanwaljit.singh@guavus.com> Closes #1657 from kjsingh/branch-1.0 and squashes the following commits: d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors

…file Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances. Author: Kanwaljit Singh <kanwaljit.singh@guavus.com> Closes #1657 from kjsingh/branch-1.0 and squashes the following commits: d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

SPARK-2641: Fixing how spark arguments are loaded from properties fil…

d8a5a12

…e for num executors

andrewor14 reviewed Sep 9, 2014
View reviewed changes

kjsingh changed the title ~~SPARK-2641: Fixing how spark arguments are loaded from properties file~~ SPARK-2641: Passing num executors to spark arguments from properties file Sep 10, 2014

asfgit closed this in 1d64812 Dec 20, 2014

Conversation

kjsingh commented Jul 30, 2014

Uh oh!

AmplabJenkins commented Jul 30, 2014

Uh oh!

pwendell commented Aug 1, 2014

Uh oh!

sryza commented Aug 1, 2014

Uh oh!

andrewor14 commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 4, 2014

Uh oh!

SparkQA commented Sep 5, 2014

Uh oh!

andrewor14 commented Sep 6, 2014

Uh oh!

SparkQA commented Sep 6, 2014

Uh oh!

SparkQA commented Sep 6, 2014

Uh oh!

vanzin commented Sep 9, 2014

Uh oh!

andrewor14 Sep 9, 2014

Choose a reason for hiding this comment

Uh oh!

kjsingh Dec 18, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Dec 20, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants