EC2 configurable workers #612
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
|
Do you mind creating a JIRA for this on the spark issue tracker? It would be good to call it something like: "Allow multiple instances per node with SPARK-EC2" |
|
Thanks - this is a nice feature. I played with this locally and it worked. |
|
I merged this. I also opened up a new JIRA to just allow us to launch multiple executors for a given app on the same node: https://issues.apache.org/jira/browse/SPARK-1706 That will provide a better general solution for users wanting to do this. |
asfgit
pushed a commit
that referenced
this pull request
May 4, 2014
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: mesos/spark-ec2#46 Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com> Closes #612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter (cherry picked from commit 4669a84) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
Thanks. |
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS Depends on: mesos/spark-ec2#46 Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com> Closes apache#612 from douglaz/ec2_configurable_workers and squashes the following commits: d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter 6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
rvesse
pushed a commit
to rvesse/spark
that referenced
this pull request
Mar 2, 2018
The names are currently used when HadoopKerberosKeytabResolverStep tries to safe the kerberos delegation token into a kubernete secret. However, the current camel case values will cause a io.fabric8.kubernetes.client.KubernetesClientException stating the following: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
When update /etc/hosts file, it should has root permation.
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Aug 18, 2023
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Aug 18, 2023
Ke 4.6.8.0 newten cherrypick * KE-41667 CDP 7.1 support hive client 3.1 (apache#612) * KE-41667 [FOLLOW UP] fix error (apache#614) * AL-8127 fix snyk, upgrade tomcat-embed-core from 9.0.68 to 9.0.72 (apache#609) AL-8127 fix snyk, upgrade tomcat-embed-core from 9.0.68 to 9.0.72 --------- Co-authored-by: jlf <longfei.jiang@kyligence.io>
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Dec 8, 2023
Ke 4.6.8.0 newten cherrypick * KE-41667 CDP 7.1 support hive client 3.1 (apache#612) * KE-41667 [FOLLOW UP] fix error (apache#614) * AL-8127 fix snyk, upgrade tomcat-embed-core from 9.0.68 to 9.0.72 (apache#609) AL-8127 fix snyk, upgrade tomcat-embed-core from 9.0.68 to 9.0.72 --------- Co-authored-by: jlf <longfei.jiang@kyligence.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS
Depends on: mesos/spark-ec2#46