Using kubectl rollout status may lead to connection churn and pauses

[briandealwis]

@npintaux is seeing long pauses with some deployments of the Cloud Code Go-Guestbook. When running with -v debug, he sees output like:

DEBU[0101] Fetching events for pod "go-guestbook-frontend-7dbd9f6695-w858j"  subtask=-1 task=DevLoop
DEBU[0101] Running command: [kubectl --context connectgateway_np-argolis-anthos_global_np-eks rollout status deployment go-guestbook-frontend --namespace default --watch=false]  subtask=-1 task=Deploy
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0149] Command output: [deployment "go-guestbook-mongodb" successfully rolled out
]  subtask=-1 task=Deploy
 - deployment/go-guestbook-mongodb is ready. [2/3 deployment(s) still pending]
 - deployment/go-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0156] Command output: [deployment "go-guestbook-backend" successfully rolled out
]  subtask=-1 task=Deploy
 - deployment/go-guestbook-backend is ready. [1/3 deployment(s) still pending]
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
 - deployment/go-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
DEBU[0282] Command output: [deployment "go-guestbook-frontend" successfully rolled out
]  subtask=-1 task=Deploy
 - deployment/go-guestbook-frontend is ready.
Deployments stabilized in 3 minutes 15.939 seconds

If you match up the Running commands with the Command outputs, you can see that the final Command output for go-guesbook-frontend at t=0282 corresponds to the kubectl rollout status at t=0101.

Nicolas reports that running kubectl rollout status from the command-line during this pause returns much faster. He's trying to isolate whether this is an issue with the remote cluster or whether it's connection-establishment cost.

Regardless, it would be better if our deployment status checks could talk to the cluster directly over a shared connection rather than repeatedly calling out to kubectl rollout status.