Adjust resource requests and autoscaling of calico-node side car containers.

[ScheererJ]

How to categorize this PR?

/area networking
/kind enhancement

What this PR does / why we need it:

Adjust resource requests and autoscaling of calico-node side car containers.

The side car containers use negligible amount of resources compared to calico-node. Therefore, requests likely just waste resources. Similarly, they fall below the minimum threshold of VPA. Hence, autoscaling them does not make sense.
minAllowed and maxAllowed of calico-node also are not that useful. Therefore, they are removed as well.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

In VPA autoscaling mode, `calico-node` should be disrupted less often as side car containers are no longer considered by VPA. Additionally, the minimum/maximum restriction are removed, which can lead to less memory consumption.

[axel7born]

/lgtm

[vlerenc]

Hmm... isn't removing a container from the VPA spec resulting in applying the default configuration, i.e. whatever is defined in spec.updatePolicy? That would keep all "removed" containers in updateMode Auto (instead of off) and even set controlledValues to RequestsAndLimits (even worse than RequestsOnly)?

cc @voelzmo

[ScheererJ]

Hmm... isn't removing a container from the VPA spec resulting in applying the default configuration, i.e. whatever is defined in spec.updatePolicy? That would keep all "removed" containers in updateMode Auto (instead of off) and even set controlledValues to RequestsAndLimits (even worse than RequestsOnly)?

cc @voelzmo

Good point. Adjusted accordingly with 0f5afa2.

[vlerenc]

Thanks @ScheererJ. I also checked in the meantime VPA and it requires the container indeed to be put explicitly into off mode if the general policy is auto.