Issues in Scaling the pods above 200 in scaled jobs

[chaithraPadmar]

Discussed in #6146

^{Originally posted by chaithraPadmar September 10, 2024}
Hi,

Version used.. 2.10.0 of keda

When we set min replica to lower numbers.. say 20
The effective max numbers of jobs post pipeline queue increases above 200 runnning jobs, we do see only 220 pods/jobs running

We have 1pipeline job per pod and using azure_pipeline triggers and scaled jobs

Is it the expected behaviour

If yes, how to manage wide range of scaling say minReplica 20, maxReplica 500

Regards
Chaithra

[chaithraPadmar]

Hi @JorTurFer ,

Kindly help with the query

Regards
Chaithra

[JorTurFer]

Hello
Yeah, min replica for ScaledJobs are the min replicas that will be available, so minReplicaCount: 20 and 100 required jobs will create 120 jobs. It's the expected behaviour and it's done to have some pods ready to process the load proactively:

[chaithraPadmar]

Hi, If i need to scale up till 350, Why MaxReplica to 350 is not working I have my config like MinReplica 20 MaxReplica 350 Its not scaling above 220, eventhough jobs or pipelines in queue Triggers are azure_triggers Regards Chaithra

…

On Wed, 2 Oct 2024, 11:59 Jorge Turrado Ferrero, ***@***.***> wrote: Hello Yeah, min replica for ScaledJobs are the min replicas that will be available, so minReplicaCount: 20 and 100 required jobs will create 120 jobs. It's the expected behaviour and it's done to have some pods ready to process the load proactively: image.png (view on web) <https://github.com/user-attachments/assets/d22dd84b-2374-4ada-8efb-61450ebc35b1> — Reply to this email directly, view it on GitHub <#6148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVEPIIRDCDW27Q2IVYLLN3ZZOHGDAVCNFSM6AAAAABN6T3ZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBXG4ZDAMRRG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorTurFer]

Have you increased the value of jobsToFetch?

[chaithraPadmar]

Yes..did a test with JobToFetch 300, 250 with multiple attempts Any relation with the number of JobToFetch and maxReplica? Do they need to have same value?? Thaks Chaithra

…

On Thu, 3 Oct 2024, 02:40 Jorge Turrado Ferrero, ***@***.***> wrote: Have you increased the value of jobsToFetch? — Reply to this email directly, view it on GitHub <#6148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVEPIJAAIAFJOVQNZF5RADZZROLJAVCNFSM6AAAAABN6T3ZFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZG4YDCOBWGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorTurFer]

You should have it bigger than the replicas. Currently, AzDO API doesn't expose the total value, so KEDA pulls all the pending jobs and filter&count them locally