Keda RabbitMQ Autoscaler Not Working and K8S API Server Timeout

[oguzhansrky]

Report

E1024 14:34:07.470495 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E1024 14:34:07.471060 1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E1024 14:34:07.471715 1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
E1024 14:34:07.472363 1 timeout.go:141] post-timeout activity - time-elapsed: 8.018879ms, GET "/apis/external.metrics.k8s.io/v1beta1" result:
E1024 14:34:07.472951 1 timeout.go:141] post-timeout activity - time-elapsed: 4.440412ms, GET "/apis/external.metrics.k8s.io/v1beta1" result:
E1024 14:34:07.473569 1 timeout.go:141] post-timeout activity - time-elapsed: 9.043293ms, GET "/apis/external.metrics.k8s.io/v1beta1" result:
W1024 14:34:12.113708 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "keda-operator.keda.svc.cluster.local:9666",
"ServerName": "keda-operator.keda.svc.cluster.local:9666",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.43.71.178:9666: connect: connection refused"
E1024 14:34:18.480827 1 wrap.go:53] timeout or abort while handling: method=GET URI="/apis/external.metrics.k8s.io/v1beta1" audit-ID="9a09bcaf-3a0e-41fc-8642-aadb2298f2c3"
E1024 14:34:18.480968 1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E1024 14:34:18.482389 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout

Expected Behavior

.

Actual Behavior

.

Steps to Reproduce the Problem

Logs from KEDA operator

2024-10-24T14:39:51Z ERROR scalers_cache error getting scale decision {"scaledobject.Name": "bug", "scaledObject.Namespace": "bug", "scaleTarget.Name": "bug", "error": "error inspecting rabbitMQ: Exception (404) Reason: "NOT_FOUND - no queue 'bug' in vhost '/'""}
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetScaledObjectState
/workspace/pkg/scaling/cache/scalers_cache.go:155
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:360
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:162
2024-10-24T14:39:51Z ERROR scalers_cache error getting scale decision {"scaledobject.Name": "bug", "scaledObject.Namespace": "bug", "scaleTarget.Name": "bug", "error": "error inspecting rabbitMQ: Exception (404) Reason: "NOT_FOUND - no queue 'bug' in vhost '/'""}
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetScaledObjectState
/workspace/pkg/scaling/cache/scalers_cache.go:155
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:360
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:162

KEDA Version

2.11.2

Kubernetes Version

1.28

Platform

Other

Scaler Details

RabbitMQ

Anything else?

I have been using Keda for a long time. Recently, the system has started to fail to respond. There are nearly 3000 scaledobjects in the system, maybe this is the cause of the problem, but I cannot find the place where it takes the timeout and I cannot intervene in Keda.