Interaction of HPA + progressDeadlineSeconds flag can cause canary failure during peak times

[marcelovcpereira]

Hi guys, we are having the following issue, currently. What is the suggested solution for this case?

Describe the issue

Flagger doesn't progress the canary while the number of running pods is below expected -> OK.
But during peak times (and also during regular canary traffic shift), a deployment with an enabled HPA can have an aggressive upscale policy in place, which constantly may change the number of expected pods due to increased traffic.
This makes flagger wait for a long time and the progressDeadlineSeconds can be triggered and fail the deployment.

To Reproduce

Start a canary deployment and keep increasing HPA number of expected pods, so flagger will wait forever until this is stabilised, eventually triggering the deadline failure.

Expected behavior

Once the pods with new version are ready, flagger should not count subsequent upscales as progressDeadlineSeconds and should just wait until pods are ready without triggering a rollback (as this was not caused by the new version).

Additional context

• Increasing the deadline can lead to slower reactions in case of real problems during pod startup
• Flagger version: 1.6.4
• Kubernetes version: 1.18
• Service Mesh provider: Linkerd
• Ingress provider: -

Thanks

[somtochiama]

Hey @marcelovcpereira ,
I am trying to reproduce this.

Start a canary deployment and keep increasing HPA number of expected pods

Do you mean I should keep increasing the hpa's spec.minReplica field during a new rollout.

[marcelovcpereira]

Hey @somtochiama , to reproduce it you need an HPA that will increase replicas depending on some variable, for example: "80% memory is reached".

Then, keep increasing the load, making your current pods memory reach 80%. HPA will spin up more pods. Then keep increasing the load, making them reach 80% again, and so on.

In each step, HPA will spin N new pods (depending on your conf) and while they are not ready, flagger pauses the rollout.

This continued load increase during a deployment is halting flagger from proceeding due to always increasing number of pods

[stefanprodan]

@marcelovcpereira can you please test #1102 and see if it solves your issue.

[marcelovcpereira]

@somtochiama @stefanprodan yes! Waiting to see this released 👍

Thank you for the work!