SPARK-2604: Incorporating memory overhead of executor during the verifcation of cluster resources, before starting the application #1571
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…fcation of cluster resources, before starting the application
|
Can one of the admins verify this patch? |
1 similar comment
|
Can one of the admins verify this patch? |
|
The memory overhead checks are actually out of date its inconsistent between yarn alpha and yarn stable as to how it works. #2253 is fixing that. I've asked them to fix the checks there also. |
|
@twinkle-sachdeva Yes it's a good idea to separate these two overheads. Part of the changes in this PR are now out-dated as of #2253, and the rest of it is incorporated in #2350. Would you mind closing this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While doing the validation for resources of application master and executor resources with cluster resources, corrected the comparison for executor resources, to keep it in sync with allocation handler, where we add and check the memory overhead too.
please see discussion https://issues.apache.org/jira/browse/SPARK-2604, to get more idea