ArgoCD argocd-application-controller sharding settings

[radiomistress]

Is your feature request related to a problem? Please describe.
https://argoproj.github.io/argo-cd/operator-manual/high_availability/ According to this document sharding is now available for argocd-application-controller.

argo-helm/charts/argo-cd/templates/argocd-application-controller/deployment.yaml

Line 20 in 1f67a85

replicas: 1

but the helm chart has the number of replicas hard coded 1. Requiring a tool other then helm to patch.

Describe the solution you'd like
While it would be trivial to just allow the replicas to be set and then to use the existing .Values.controller.env. since you need to set the replicas and the ENV variable I think it would be a better experience and a more straight forward experience to have a dedicated .Values.controller.shards setting which would control both the ENV variable and then number of replicas.

Describe alternatives you've considered
The alternative is to just allow the number of replicas to be set. But It could allow for a scenario where someone bumps the replicas expecting better performance but without the ARGOCD_CONTROLLER_REPLICAS environment variable being set I think it would be better to tie both variables together.

[radiomistress]

I take it back setting the replicas is easier from a maintenance perspective, #496 adding some comments and an example for the ENV variable I think is sufficient.

[okdas]

In the HA doc page the sharding example includes a StatefulSet, though this chart manages the controller with Deployment. I wonder if this is necessary for sharding.

[radiomistress]

I have not found that to be the case. While StatefulSet may be a better pick for this part of ArgoCD I have not found any StatefulSet features in use, To my knowledge the only reason one would pick StatefulSet over deployment is volumeClaimTemplates which this service doesn't use. I just had to add more replicas and pass in the environment variable and it looked to be working as expected. I think without volumeClaimTemplates being apart of the this, deployment and statefulset can be interchanged without issue.

[limoges]

For this to work you need to switch to a StatefulSet.

Otherwise we get this:
hostname should ends with shard number separated by - but got: argocd-application-controller-64b4d44dbc-bx9m7

[limoges]

Using the update from #496 introduces a bug with image tag v1.8.0-rc1

Edit:
See how they shard:
https://github.com/argoproj/argo-cd/blob/303925f4a016d7fee80bbe357341df7d3febe298/controller/sharding/sharding.go#L20

[limoges]

For me, the question becomes how to make the transition to StatefulSet with the least amount of problems for user?

The simplest approach is to just "simply" translate the Deployment into a StatefulSet with adjustments.
Unfortunately, that would create controller downtime. That's not a huge issue for our monorepo using around 800 applications on 7 clusters.

Community-wise, this is a common issue with charts (upgrades are rarely safe for high-availability contexts).

[radiomistress]

Make sense to me. I can work on switching it over to a StatefulSet.

[yydzhou]

Make sense to me. I can work on switching it over to a StatefulSet.

@mr-sour any progress on this?

[radiomistress]

I'm fast but not that fast 😆 should be done and tested by end of day Friday.

[radiomistress]

@limoges on the note of upgrading from deployment to statefulset, What I did was sync the new statefulset without pruning deployment, I then pruned the deployment and I didn't experience any downtime. Granted for a few minutes your running 2 controllers that aren't aware of each other so that could have side-effects that I didn't see but it doesn't look like anything catastrophic happened.

[github-actions]

Stale issue message