[SPARK-8728] Add configuration for limiting the maximum number of active stages in a fair scheduling queue #7119
Conversation
|
Can one of the admins verify this patch? |
| scheduleTaskAndVerifyId(4, rootPool, 1) | ||
| scheduleTaskAndVerifyId(5, rootPool, 2) | ||
|
|
||
| verifyNoRemainedTask(rootPool) |
There was a problem hiding this comment.
Please rename this to noTasksRemain, refactor that method slightly to produce the Boolean nextTaskSetToSchedule.isEmpty, then change this line to assert(noTasksRemain).
There was a problem hiding this comment.
Better yet, make it noTasksRemainIn(pool: Pool).
|
@kayousterhout @markhamstra is this something we want? |
|
Given that there is not sufficient interest in this feature, @sirpkt can you close this PR? |
|
@andrewor14 I'm interested; was actually thinking of "maxShare" to go with the min. Lower-weighted pools will still consume the entire cluster when others are idle, and must finish processing partitions before shares can be returned to the higher weight. It makes a lot of sense to be able to restrict a pool and leave capacity immediately available for the rest of the cluster. |
It only takes predefined number of Schedulables in getSortedTaskSetQueue.
'maxRunning' is added in fair scheduler configuration to limit the maximum number of concurrently running stages.
If 'maxRunning' is not set, no limitation is applied.