Apache Kafka limitToPartitionsWithLag is not respected

[kamilglod]

Report

When limitToPartitionsWithLag is set to true and have some partitions without lag keda is still keeping max number of pods scaled

Expected Behavior

Keda will scaled up to the number of partitions with lag

Actual Behavior

Keda is scaling to max number of pods

Steps to Reproduce the Problem

1. Have keda scaledobject with limitToPartitionsWithLag = true
2. Have a lot of events on topic distributed unevenly on all partitions that would result in scaling max number of pods
3. Observe that when keda will start MAX number of pods instead of number of partitions - partitions without lag

Logs from KEDA operator

2024-06-12T08:10:21Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "75c29f13-3068-455d-acc1-6b228ed37030"}
2024-06-12T08:10:36Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "7b0ac5eb-f1a4-4460-857d-8e7204cdecd9"}
2024-06-12T08:10:52Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "a1adb2b8-636e-4dcc-8e92-e87421976070"}
2024-06-12T08:11:07Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "5b899b41-a293-4ca8-8553-bc18f849c70b"}
2024-06-12T08:11:22Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "9566c55b-f2bb-473b-888a-4faf38e3c1c3"}
2024-06-12T08:11:38Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "e0739a9b-1ca1-44b2-af44-7c43b1d5826d"}
2024-06-12T08:25:02Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "dd7e3f55-bd64-4ab9-9a2e-0bd26bf0ebea"}
2024-06-12T08:26:35Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "0048c21e-834d-418f-8ce0-a2443cc5c575"}
2024-06-12T08:26:50Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "52a36446-e4ff-48c6-9b27-c75283a69c2f"}
2024-06-12T08:27:06Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "93cbee63-53c1-40ff-afda-996aaeb0bf4e"}
2024-06-12T08:27:21Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "d3a84b4a-7f67-481b-8d75-4bc5f8e27d96"}
2024-06-12T08:27:37Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "90e07ab4-0f35-416d-8340-b950d211345b"}
2024-06-12T08:27:52Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "6d524afb-5954-47cb-a3c3-19ebd3d3b03f"}
2024-06-12T08:29:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "14b6c5f0-b0e3-44d0-a680-a663006b39e9"}
2024-06-12T08:29:25Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "ab86f102-1dbb-4e0a-92df-ddfdc08edc19"}
2024-06-12T08:30:57Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"fis-process-offer-scaled-object","namespace":"data-integration"}, "namespace": "data-integration", "name": "fis-process-offer-scaled-object", "reconcileID": "3e3112db-65c3-4b9e-9558-421269e74e7e"}

KEDA Version

2.12.0

Kubernetes Version

1.28

Platform

Google Cloud

Scaler Details

Kafka

Anything else?

Scaled object definition

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  labels:
    k8s.prisjakt.se/app: feed-ingestion-scaled-object
    k8s.prisjakt.se/owner: team-data-operations
    k8s.prisjakt.se/tier: backend
    team: ingestion
    component: feed-ingestion-service
    environment: dev
  name: fis-process-offer-scaled-object
  namespace: data-integration
spec:
  scaleTargetRef:
    name: fis-process-offer
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 60
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: xxx
        consumerGroup: xxx
        topic: xxx
        lagThreshold: '10000'
        offsetResetPolicy: earliest
        tls: enable
        sasl: plaintext
        limitToPartitionsWithLag: "true"
      authenticationRef:
        name: kafka-trigger-auth

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[kamilglod]

I'm still waiting for an answer here.

[dttung2905]

Hi @kamilglod,
Sorry for the late reply. Could you try to update to the latest version ?
I see that you are using KEDA version 2.12 . However, limitToPartitionsWithLag is only introduced in version 2.13 and above
in #5060.
You can double check the documentation here as well https://keda.sh/docs/2.12/scalers/apache-kafka/

[kamilglod]

@dttung2905 I can confirm that it works well with 2.13, sorry for bothering.

[dttung2905]

@kamilglod awesome. Glad you get it resolved