External scaler goes to CrashloopBackoff
state frequently
#783
Unanswered
IsuruBoyagane15
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have been using the add-on to scale my deployments in a AKS cluster.
I have multiple workloads to be scaled. Generally, the system works as expected.
But, I see the
external-scaler
pod goes toCrashloopBackoff
state frequently.When I checked the log in the
external-scaler
, I found this error log.{"level":"error","ts":1694510987.376597,"caller":"scaler/main.go:69","msg":"creating a queue pinger","error":"requesting the queue counts from http://10.0.7.206:9090/queue: Get \"http://10.0.7.206:9090/queue\": dial tcp 10.0.7.206:9090: i/o timeout","errorVerbose":"Get \"http://10.0.7.206:9090/queue\": dial tcp 10.0.7.206:9090: i/o timeout\nrequesting the queue counts from http://10.0.7.206:9090/queue\ngithub.com/kedacore/http-add-on/pkg/queue.GetCounts\n\t/workspace/pkg/queue/queue_rpc.go:76\nmain.fetchCounts.func1\n\t/workspace/scaler/queue_pinger.go:222\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"main.main\n\t/workspace/scaler/main.go:69\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
As I understood, the
external-scaler
could not connect to theadmin-service
of oneinterceptor
pod. (interceptor
had 3 replicas in the deployment;10.0.7.206
is the pod IP of one of the replicas)I tried roll-out restarting
external-scaler
andinterceptor
pods. No luck.During this time, the deployments that are supposed to be scaled to zero after the cool-down period (5 mins), were not scaled to zero in 5 mins.
Finally, I scaled
interceptor
deployment from 3 to 1. With that change, after some time, the setup started to work as expected.Any possible reason for this behavior?
Beta Was this translation helpful? Give feedback.
All reactions