[SPARK-2140] Enabling YARN memory overhead calculation from heap of the JVM #1906
Conversation
|
Can one of the admins verify this patch? |
copester
changed the title
| val minResMemory: Int = newApp.getMinimumResourceCapability().getMemory() | ||
| var amMemory = ((args.amMemory / minResMemory) * minResMemory) + | ||
| ((if ((args.amMemory % minResMemory) == 0) 0 else minResMemory) - | ||
| memoryOverhead ) | ||
| args.amMemory |
There was a problem hiding this comment.
There is always returns args.amMemory.
|
Jenkins, this is ok to test |
|
QA tests have started for PR 1906 at commit
|
|
QA tests have finished for PR 1906 at commit
|
|
@copester Any chance you can upmerge this and update based on the jira conversation. I think the issue is actually with yarn alpha version now. The yarn alpha version of this shouldn't subtract out the memoryOverhead since its using that directly as the heap size and adding in the memory overhead when it requests the AM container ( memoryResource.setMemory(args.amMemory + memoryOverhead)). We also don't really need to do the logic there of rounding based on the getMinimumResourceCapability because Yarn will handle that for us. So I think we can actually remove the calculateAMMemory routines and just use args.amMemory. |
|
I assume this is replaced by #2253. Can you close this one? |
No description provided.