[LIVY-870] Adding changes to provide Spark config using --properties-file #331
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akshatb1
changed the title
|
@akshatb1 would you please rebase to latest master to fix the CI failure? |
akshatb1
force-pushed
the
spark-submit-properties
branch
from
e5b8f0c to
7df3b4e
Compare
…oad on Yarn and Livy" This reverts commit dc867be.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently for batch jobs, individual Spark configs are added as --conf parameter in Spark submit command line. If there are many Spark configurations added in Livy APIs, it might increase the length of command line and in some cases might cause issues due to limit on command line length (Windows has 8191 character length limit on the command line length.).
In this PR, modifying the behavior to merge all the configurations into a temporary properties file and adding that as --properties-file opt in spark-submit command
JIRA: https://issues.apache.org/jira/browse/LIVY-870
How was this patch tested?
Verified Spark job submission in a local Yarn cluster. Checked in the logs that all configurations are merged to a properties file which is added as an opt in spark-submit command. Please see the logs below.