Only update kops-controller pods on deletion

[olemarkus]

This will in most cases mean kops-controller is only updated when the CP node is rolled and should increase the chances of compatible controller<->CP updates

[hakman]

/cc @justinsb @johngmyers

[hakman]

kops-controller uses leader election to decide the leader. Just noticed that the mechanism changed there from ConfigMaps locking to Lease API. Update on delete would break upgrading to the new mechanism.
kubernetes-sigs/controller-runtime#1144

Not sure what the plan is long term, but having new and old kops-controller at the same time in the cluster may generate unexpected results depending on how old the leader is and what are the tasks that it has to do.

[olemarkus]

The only controller that kops-controller is running is node labeller. This one only does idempotent operations and can run perfectly fine in parallel. For CP nodes, it already fights with protokube.

Running rolling-update of kops-controller will result in similar conflict, only for a (much) shorter duration.

[johngmyers]

The leader election method change would also be an issue for the status quo, just for the shorter amount of time that it takes for the DaemonSet controller to do its rolling update.

/lgtm
/approve
/hold for any last-minute objections

[rifelpet]

@olemarkus can you rebase this?

[olemarkus]

/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johngmyers, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johngmyers,rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[olemarkus]

/retest