intermittent network failures during rollout promotions

[shaneutt]

Current Behavior

When performing a rollout while hitting the Kong Gateway with a curl bomb, as the promotion switches between the blue and green deployment I consistently receive intermittent network failures. In my testing the failures were all within about 1 second.

Expected Behavior

I would expect a smooth transition with no failures during promotion.

Steps To Reproduce

1. create a ktf cluster (with metallb)
2. deploy the CRDs and KGO
3. deploy the config/samples/dataplane.yaml example DataPlane
4. once ready, run a curl bomb in the background (e.g. while true; do curl ${SVCIP}; done)
5. run kubectl edit dataplanes dataplane-example and change the image
6. once ready add the promotion annotation, and witness several rounds of curl failures

Operator Version

v1.1.0