nginx-ingress controller with autoscaler enabled, immediately scale up to maximum replicas amount #10178
Comments
k8s-ci-robot
added
needs-triage
|
/remove-kind bug show |
k8s-ci-robot
added
triage/needs-information
|
Sure: |
|
This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach |
github-actions
bot
added
the
lifecycle/frozen
|
We have the same problem with our nginx-ingress deployment on AKS (however we use Standard_B2ms machines). I wonder if the autoscaling feature necessarily needs resource limits to be set so it can evaluate what exactly 50% or 80% of cpu/memory usage is. values.yaml: |
|
Has this been fixed? I deployed nginx via helm a few days ago with this config: And only one pod was created. In another cluster when I've done this in the past it created two. |
|
/remove-triage needs-information |
k8s-ci-robot
added
kind/bug
|
/remove-lifecycle frozen |
k8s-ci-robot
removed
the
lifecycle/frozen
|
Hi, controller in idle state use ~60/120Mi of memory when you got and enabled autoscaling with default 50% targetMemoryUtilizationPercentage hpa will always scale up pods to maxReplicas |
What happened:
Autoscaling seems to scale to maximum capacity as soon as the ingress controller is deployed.
What you expected to happen:
Not seeing the ingress scale immediately.
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
NGINX Ingress controller
Release: v1.8.1
Build: dc88dce
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
Kubernetes version (use
kubectl version):Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"94c50547e633f1db5d4c56b2b305670e14987d59", GitTreeState:"clean", BuildDate:"2023-06-12T18:46:30Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Environment:
uname -a): Linux nginx-ingress-ingress-nginx-controller-7c9f44b5f8-z5z24 5.15.0-1040-azure Fix the hyperlink in CONTRIBUTING.md for issue search #47-Ubuntu SMP Thu Jun 1 19:38:24 UTC 2023 x86_64 Linuxkubectl version: See abovekubectl get nodes -o wide:helm ls -A | grep -i ingresskubectl describe ingressclasseskubectl -n <ingresscontrollernamespace> get all -A -o widekubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>kubectl -n <appnnamespace> get all,ing -o widekubectl -n <appnamespace> describe ing <ingressname>If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
Others:
kubectl describe ...of any custom configmap(s) created and in useHow to reproduce this issue:
This issue has been tested and reproducible 100% of the time on Azure Kubernetes Service
First step would be do deploy an AKS service with nodes Standard_B8ms or higher. Lower class nodes don't seem to have this problem.
Simply enable scaling when installing the chart and you should see the behaviour reported.
helm install nginx-ingress ingress-nginx/ingress-nginx --set controller.autoscaling.enabled=trueAnything else we need to know:
I encountered this anomaly on a cluster with B12ms VMs used as nodes and started testing possible causes. Noticed that up until B4ms this does not happen. I can't figure out why the exact same configuration misbehave on nodes with VM with more memory available but what seems to happen is that the pod is deployed and as soon as I can get some metrics out, it's RAM usage is > 80% which trigger the deployment of new replicas.
The initial cluster I have made aware of the issue had only nginx, cert-manager, grafana, prometheus and loki deployed.
After some consideration and experiments I deployed a new cluster from scratch and deployed only nginx-ingress in it via helm, the behaviour kept happening in the same way.
I tried increasing/lowering maxReplica and it always deploys all available replicas.
I also tried enabling the explicit scaleUp and scaleDown policies in the chart, still, it only increases the amount of replicas and never scale them down.
During my tests I also tried using the two previous version of the helm chart (corresponding to the app versions 1.7.1 and 1.8.0) and the behaviour was the same.
If someone can check if it's happening with other cloud providers when using similar hardware for the nodes, it could help understanding if I need instead to go to Microsoft asking for clarifications.
The text was updated successfully, but these errors were encountered: