SQS Autoscaler sum of multiple queues

[yogevyuval]

Hi,

Is it possible with Keda to have an SQS autoscaler that is based on multiple queues? Specifically it would be very helpful to have it based on the sum of queues or the average of the number of messages in the queues.

[tomkerkhove]

You can use multiple triggers in the same scaledobject, but it will start scaling as soon as 1 of them is matching the criteria, so that's not a solution for you then?

[yogevyuval]

You can use multiple triggers in the same scaledobject, but it will start scaling as soon as 1 of them is matching the criteria, so that's not a solution for you then?

It's a start, but it's not an optimal solution for my situation.
Let's say you have a consumer that consumes from 3 different types of input queues, you want the consumer to scale accordingly. It's a different state when 1 queue has 1000 messages and the other ones are empty, than if all of the queues have 1000 messages each. The target scaling number would be really different in that case

[yogevyuval]

I see that @TsuyoshiUshio added the abillity to choose multipleScalersCalculation on ScaledJobs, is that possible on ScaledObject as well?

[zroubalik]

It might be an option. What I see is that in this case we would do the calculation from mutliple metrics on KEDA side and expose just the final metric to the HPA. It's definitely an idea worth exploring.

[Wirone]

Was this explored and implemented? We're currently doing PoC of new workers approach with Symfony Messenger and KEDA. Messenger can consume from multiple exchanges/queues and we want to group queues by their characteristics (e.g. many queues with small, from-time-to-time traffic), then scale it with KEDA. However, during PoC I see that if I have multiple triggers, they are handled separately while in our case it would be better if it could work on sum of queued messages (like described here).

[joebowbeer]

@yogevyuval Upvote request for custom formula here: #2440

[joebowbeer]

Addressed by custom formula?

https://keda.sh/docs/2.15/concepts/scaling-deployments/#scaling-modifiers

[yogevyuval]

Addressed by custom formula?

https://keda.sh/docs/2.15/concepts/scaling-deployments/#scaling-modifiers

I think this works! Thank you, closing this.