Add toleration for new "control-plane" node label for Concierge deploy

[cfryanr]

Addresses #1010.

Add toleration for new "control-plane" node label for Concierge deployment, which will be required instead of old "master" name for Kubernetes 1.24+.

Release note:

None, as this change won't be noticeable by users. See comment below for more information.

NONE

[codecov]

Codecov Report

Merging #1031 (8dc4a89) into main (619b8c1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1031   +/-   ##
=======================================
  Coverage   79.38%   79.38%           
=======================================
  Files         133      133           
  Lines        9972     9972           
=======================================
  Hits         7916     7916           
  Misses       1786     1786           
  Partials      270      270           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 619b8c1...8dc4a89. Read the comment docs.

[cfryanr]

I confirmed that even without this PR, the Concierge deploys fine on multi-node clusters using the pre-release of Kubernetes 1.24. The Concierge does not ask to be scheduled on control plane nodes (no affinity for control plane), but it does tolerate control plane nodes. Therefore, in a multi-node cluster that has enough resources, the Concierge will be scheduled onto a worker node and the Concierge tolerations don't matter. On a single-node cluster, the single node would not be tainted as "control-plane", or else most apps would fail to deploy, so again the Concierge tolerations don't matter.

Note that the "concierge-kube-cert-agent" pod does want to be scheduled on the control plane nodes, but that code already correctly handles the new "control-plane" taint name, so there is no change needed in that code.

Theoretically, if a multi-node cluster were suffering from a lack of resources and the Concierge could only be scheduled (or rescheduled) on to a control plane node, then we would want to allow that by tolerating the "control-plane" taint. That would be preferable to losing the Concierge pods. For this reason, this PR seems like it still has some value.

[margocrawf]

This approach makes sense to me, particularly since if anyone feels really strongly that the concierge should not run on their control plane nodes, they are free to edit the deployment.yaml, right?

I can't think of a reason we should outright avoid running on control plane nodes.

[enj]

Conceptually, the concierge is part of the control plane. The functionality it provides is what we wished the API server supported directly anyway.

I could see arguments for making the concierge have affinity for the control plane nodes, but we do not need to worry about that right now.

A similar line of reasoning could be applied to the supervisor.