Deadlocks configuring envoy for IAP

[jlewi]

I'm noticing deadlocks and other problems configuring envoy using the IAP script. Some of the problems I observe

1. iap.sh is never able to acquire the lock and therefore able to write the envoy-config.json
2. envoy container is crash looping - prevents GCP loadbalancer from detecting the backend is heathy

I think we should make the following changes

1. There should be a single pod responsible for enabling IAP and updating the envoy-config map as needed

   • We should move this out of the sidecar and into a separate deployment
   • Locking should be less important because there won't be contention

2. The envoy sidecars are now just responsible for updating envoy config based on the config map

   • They no longer need to acquire a lock
   • They can periodically check the config map and compute a hash to know when it changes

3. We should provide a default config that will allow envoy to startup but ensure non secure traffic is blocked

   • This way we can avoid the problems with the ingress thinking the backend is unhealthy.

[jlewi]

/assign @jlewi

[jlewi]

Closing this issue because #924 should fix the deadlocks and we have open issues for the other issues.