Allow activationQueueLength (SQS) to respect in-flight Messages

[claudio-vellage]

Proposal

We want to be able to tell KEDA to not kill the workers if the in-flight messages are above the activationQueueLength.

Use-Case

We have a special use-case, where we need to process a certain amount of messages, before we can eventually remove those messages from the queue. Therefore, we have a lot of in-flight messages, but unfortunately KEDA scales down our worker, before those in-flight messages are processed, leading to a lot of messages piling up in the dead letter queue. How can we achieve our goal, so that KEDA will consider the in-flight messages too for activationQueueLength or is it already the case and our issue is elsewhere? Let me know if you need more information about what exactly we are trying to achieve.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

[tomkerkhove]

Reasonable ask, we'll need to wait for somebody to contribute this.

[joebowbeer]

@claudio-vellage asks:

How can we achieve our goal, so that KEDA will consider the in-flight messages too for activationQueueLength or is it already the case and our issue is elsewhere?

It is already the case unless you set scaleOnInFlight=false.

By default, scaleOnDelayed is false and scaleOnInFlight is true.

This means that both sources (normal + inflight) are counted.

Then the activationQueueLength is compared to whatever length is counted above.

https://github.com/kedacore/keda/blob/main/pkg/scalers/aws_sqs_queue_scaler.go#L137

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.