Support for HA in Rollouts operator

[jgwest]

Currently, RolloutManager has no way to set the number of replicas for the Argo Rollouts controller (part of Argo Rollouts install). The ability to increases the number of replicas is required for HA install. Rollouts supports leader election and multiple pods are supported in an Active-Passive fashion.

So while multiple replicas do not benefit performance, it is useful for HA purposes for situations like when a node goes down for service.

Following the same .spec as our sibling operator, Argo CD operator, we can add a new 'ha' field to RolloutManager:

kind: RolloutManager
metadata:
  name: my-rollouts-install
spec:
  # (...)
  ha:
    enabled: # true/false
    # (If we get a request for additional customization of HA fields, such as anti-affinity rules, we can add them into .spec.ha)  

Issue Criteria:

• Update RolloutManager struct to add new field
• When .spec.ha is enabled:
  • The --leader-elect=true flag should be passed as a parameter to the Rollouts Deployment.
    • True is already the default for this parameter when it is not specified, but ensuring it is always set for this case can't hurt.
  • The number of replicas of the Deployment should be 2 (rather than 1)
  • Anti-affinity rules are set
• As above, we should probably also add default anti-affinity rules when .spec.ha is enabled:
  • This will take some investigation as I am personally not familiar with best practices for anti-affinity rules. I would recommend looking at how Argo CD operator and Argo CD setup their affinity rules.
  • Similar to Argo CD operator, we will not allow users to configure anti-affinity rules at this time. Instead, we will just set some useful default anti-affinity rules.
    • In the future if get a specific request here, it is easy to add.
• Unit/E2E tests that verify when .spec.ha.enabled is true, that the Rollouts Deployment has the expected values.

[jgwest]

Red Hat External JIRA: https://issues.redhat.com/browse/GITOPS-5622