Reduce VM CPU overhead by 20%

[tstromberg]

Related to #3207 and #5398

#3207 can probably be closed - I don't believe we are at 50% usage any longer.

[priyawadhwa]

Running pidstat 1, I noticed that every 5 seconds the following command results in a high uptick in CPU usage (around 80%, from ~20%, for one core) on my linux machine:

/usr/local/bin/kubectl apply -f /etc/kubernetes/addons -l kubernetes.io/cluster-service!=true,addonmanager.kubernetes.io/mode=Reconcile --prune=true --prune-whitelist core/v1/ConfigMap --prune-whitelist core/v1/Endpoints --prune-whitelist core/v1/Namespace --prune-whitelist core/v1/PersistentVolumeClaim --prune-whitelist core/v1/PersistentVolume --prune-whitelist core/v1/Pod --prune-whitelist core/v1/ReplicationController --prune-whitelist core/v1/Secret --prune-whitelist core/v1/Service --prune-whitelist batch/v1/Job --prune-whitelist batch/v1beta1/CronJob --prune-whitelist apps/v1/DaemonSet --prune-whitelist apps/v1/Deployment --prune-whitelist apps/v1/ReplicaSet --prune-whitelist apps/v1/StatefulSet --prune-whitelist extensions/v1beta1/Ingress --recursive 

the addon manager currently polls every 5 seconds.

@tstromberg, WDYT of increasing polling time to 20 or 30 seconds? I have a graph showing CPU usage for the qemu process at 5 seconds and 20 seconds
for addon manager poll time.

[sharifelgamal]

This is being targeted for early next year.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[priyawadhwa]

/remove-lifecycle rotten