[Question] Can startupPolicy set as LeaderReady meet the requirement of gang scheduling?

[JesseStutler]

Hi community, currently I'm researching the lws, I found that there is a field called startUpPolicy in lws spec. Currently, the sample yamls of lws in this repo have not written this field. Is there such a scenario? I am wondering if the Leader Pod needs to run first, and then create worker statefulset, whether the Leader Pod + n Workers Pods (a PodGroup) are needed to meet gang scheduling. Imagine that if the cluster resources are limited at this time and the gang cannot satisfy, then the workers pods cannot be scheduled, but the Leader Pod has already occupied the resources in the cluster. So I am wondering whether this scenario conflicts with the original intention of gang, or whether this scenario still requires gang scheduling? Hope to get some feedback, thanks!

[kerthcet]

I think gang is important to make sure all the pods belong to the same PodGroup (lws replica) Should Be schedulable, then we'll schedule the leader pod first, following the workers. But one potential concern is we need to create separate PodGroup for each replica.

[JesseStutler]

I think gang is important to make sure all the pods belong to the same PodGroup (lws replica) Should Be schedulable, then we'll schedule the leader pod first, following the workers. But one potential concern is we need to create separate PodGroup for each replica.

Yes, I know, I'm trying to create a separate podgroup for lws replica in volcano, so then it can meet gang scheduling. However, when combining them into a PodGroup like this, the starting point of LeaderReady seems to conflict with gang scheduling, because the Leader Pod needs to be scheduled first

[kerthcet]

Yes, they're conflict with each other right now, we may take it as an useful input for gang design. Or do you have any insight fro m volcano perspective?

Also see discussion here: #167

[JesseStutler]

Yes, they're conflict with each other right now, we may take it as an useful input for gang design. Or do you have any insight fro m volcano perspective?

Also see discussion here: #167

Thanks. I will update here anytime when I have some opinions

[kerthcet]

You can hard-coded the startup Policy for now, it will not impact the final result, but the workers will struggle to connect to the leader for several times until the leader is ready.

[JesseStutler]

@kerthcet My current modification is that minMember is 1, but the resource request for the Group is still the sum of the resources of the Leader and all Worker Pods. If the cluster has enough resources for this Group, the Leader Pod will be scheduled first. However, if the cluster does not have enough resources for this Group, then this Group cannot enter the scheduling queue. But this approach also cannot safely guarantee gang scheduling, because the leader pod and worker pods are scheduled in two rounds. It is very likely that the leader pod is successfully scheduled at first, but later, when scheduling the worker pods, a pod with higher priority is scheduled. Therefore, reserving resources for this Group by the scheduler might be a better strategy. Please take a look at my design for LeaderReady in this PR: #407. I hope the upstream community can accept my proposal and allow me to proceed with integrating lws into the custom scheduler and make lws better.