| title | description | keywords | services | documentationcenter | author | manager | editor | ms.assetid | ms.service | ms.workload | ms.tgt_pltfrm | ms.devlang | ms.topic | ms.date | ms.author |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
How do I configure Spark application through spark-submit on HDInsight clusters? | Microsoft Docs |
Use the Spark FAQ for answers to common questions on Spark on Azure HDInsight platform. |
Azure HDInsight, Spark, FAQ, troubleshooting guide, common problems, application configuration, spark-submit |
Azure HDInsight |
na |
arijitt |
0DBF3DBF-FD6C-4F87-A8AE-7F1D74B094A2 |
multiple |
na |
na |
na |
article |
03/30/2017 |
arijitt |
Need to configure at submit time through spark-submit, the amount of memory and number of cores that a Spark application can use on HDInsight clusters.
-
Refer to the topic Why did my Spark application fail with OutOfMemoryError? to determine which Spark configurations need to be set and to what values.
-
Launch spark-shell with a command similar to the following (change the actual value of the configurations as applicable):
spark-submit --master yarn-cluster --class com.microsoft.spark.application --num-executors 4 --executor-memory 4g --executor-cores 2 --driver-memory 8g --driver-cores 4 /home/user/spark/sparkapplication.jar