KEDA operator restarting many times

[kevinmat]

Hi,

I am using KEDA version 2.0.0 , KEDA operator POD and KEDA metric pod are running on Kubernetes. I have defined many Kafka Triggers.
However I have noticed that the KEDA operator POD restarts many times and when I see the error in the pod logs of KEDA operator I see the following:

"error": "error getting scaler for trigger #0: error creating kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:184
github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:135
2021/01/31 13:14:10 http: Accept error: accept tcp [::]:8081: accept4: too many open files; retrying in 1s

Could anyone please suggest how to debug this issue . I have around 700 Kafka topics which have triggers associated across 5-6 of my K8s applications which I wanted to auto scale based on kafka lagThreshold and use KEDA.

Thanks in advance,
Kevin

[zroubalik]

@kevinmat could please try KEDA 2.1? We have made some performance improvements in that release, wrt to a bigger number of triggers.

[kevinmat]

@zroubalik Thanks for your reply. As you said I moved to KEDA 2.1. However even with the latest version as I increase the number of Kafka topics and scalar definitions I run into errors like below in KEDA metrics server pod
I tried with 50 Kafka topics each 5 partitions and get errors

E0201 14:39:12.505766 1 writers.go:120] apiserver was unable to write a fallback JSON response: http2: stream closed

I0201 14:39:15.297546 1 streamwatcher.go:117] Unable to decode an event from the watch stream: context deadline exceeded (Client.Timeout or context cancellation while reading body)

As a result of the above , scaling does not happen.
Is there any KEDA setting which I would need to do?

[kevinmat]

I also see the following printed in the Metrics Logs , Is it that HPA keeps invoking the KEDA metrics server for each metric in a loop?

I noticed that The Listing keeps increasing with topics at some point start getting the errors which I mentioned above

I0201 14:40:11.735236 1 trace.go:205] Trace[1032612154]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-sample/kafka-kevin_test_35-testsample,user-agent:kube-controller-manager/v1.18.10 (linux/amd64) kubernetes/62876fc/system:serviceaccount:kube-system:horizontal-pod-autoscaler,client:10.76.33.130 (01-Feb-2021 14:39:14.009) (total time: 57725ms):
Trace[1032612154]: ---"About to List from storage" 0ms (14:39:00.009)
Trace[1032612154]: ---"Listing from storage done" 57725ms (14:40:00.735)