feat: Support appset in any namespace ArgoCD feature

[svghadi]

What type of PR is this?

/kind enhancement

What does this PR do / why we need it:
ApplicationSet in any namespace feature allows users to mange ApplicationSet resources in namespaces other than the ArgoCD/control plane's namespace This PR adds support to configure this Argo CD feature via operator.

Two new fields are introduced in ArgoCD CR to support ApplicationSet in any namespace.

• .spec.applicationSet.sourceNamespaces - list of namespaces to allow users to create appset resources
• .spec.applicationSet.scmProviders - optional list of scm providers

Changes:

• Add new applicationset-controller clusterrole & clusterrolebinding for cluster-scoped ArgoCD instance
• Add new role & rolebinding in target namespace for applicationset-controller
• Update role for argocd-server in target namespace to grant access to appsets
• Add new fields under .spec.applicationSet in ArgoCD CR
• Modify applicationset-controlller deployment command to include new params to support appset in any ns
• Add unit & kuttl test

Important Note:

• This feature can only be enabled and used when Argo CD is installed as a cluster-wide instance
• ApplicationSet in any namespace requires App in any namespace configured to function properly. For instance, the applications generated by applicationset in namespace foo won't be created if App in any namespace for foo is not configured. How it would look in ArgoCD CR is

  spec:
    applicationSet:
      sourceNamespaces:
        - foo
    # appset in any ns requires apps in any ns to function properly
    sourceNamespaces:
      - foo

• ApplicationSet in any namespace doesn't support default SCM providers list. SCM providers are disabled by default when this feature is enabled via operator, admins need to explicitly defined the allowed SCM providers if required. Allow ApplicationSets in any namespace without setting allowed-scm-providers argoproj/argo-cd#15246

Have you updated the necessary documentation?

• Documentation update is required by this PR.
• Documentation has been updated.

How to test changes / Special notes to the reviewer:
Automated kuttl test:

kubectl kuttl test tests/k8s/ --config tests/kuttl-tests.yaml  --test 1-002_validate_cluster_config
kubectl kuttl test tests/k8s/ --config tests/kuttl-tests.yaml --test 1-037_validate_applicationset_in_any_namespace

Manual test:

1. Create namespaces for testing

   kubectl create ns test
   kubectl create ns apps
   kubectl create ns dev
   kubectl create ns prod
   kubectl label namespace dev argocd.argoproj.io/managed-by=test
   kubectl label namespace prod argocd.argoproj.io/managed-by=test

2. Checkout the code & run the operator

   ARGOCD_CLUSTER_CONFIG_NAMESPACES=test make install run

3. Create a ArgoCD in test to make it cluster-scoped

   # argocd.yaml
   apiVersion: argoproj.io/v1beta1
   kind: ArgoCD
   metadata:
     name: argocd
     namespace: test
   spec:
     applicationSet:
       sourceNamespaces:
         - apps
     # appset in any ns requires apps in any ns to function properly
     sourceNamespaces:
       - apps

4. Validate creation of applicationset-controller RBAC resources

   $ kubectl get argocd/argocd -n test -o jsonpath='{.status.applicationSetController}'
   Running
   
   $ kubectl get clusterrolebinding argocd-test-argocd-applicationset-controller
   NAME                                           ROLE                                                       AGE
   argocd-test-argocd-applicationset-controller   ClusterRole/argocd-test-argocd-applicationset-controller   2m44s
   
   $ kubectl get rolebindings -n apps
   NAME                         ROLE                              AGE
   argocd_apps                  Role/argocd_apps                  91s
   argocd-test-applicationset   Role/argocd-test-applicationset   90s
   

5. Edit default AppProject to allow apps-in-any-ns to use it

   # kubectl edit appproject default -n test
   apiVersion: argoproj.io/v1alpha1
   kind: AppProject
   metadata:
     name: default
     namespace: test
   spec:
     sourceNamespaces:
     - apps

6. Create an applicationset in allowed apps namespace

   # appset.yaml
   apiVersion: argoproj.io/v1alpha1
   kind: ApplicationSet
   metadata:
     name: guestbook
     namespace: apps
   spec:
     generators:
       - list:
           elements:
             - cluster: engineering-dev
               url: 'https://kubernetes.default.svc'
               ns: dev
             - cluster: engineering-prod
               url: 'https://kubernetes.default.svc'
               ns: prod
     template:
       metadata:
         name: '{{cluster}}-guestbook'
       spec:
         project: default
         source:
           repoURL: 'https://github.com/argoproj/argo-cd.git'
           targetRevision: HEAD
           path: 'applicationset/examples/list-generator/guestbook/{{cluster}}'
         destination:
           server: '{{url}}'
           namespace: '{{ns}}'

7. Verify that applicationset and applications are generated in allowed apps namespace

   $ kubectl get applicationset/guestbook -n apps
   NAME        AGE
   guestbook   22s  
   
   $ kubectl get applications -n apps
   NAME                         SYNC STATUS   HEALTH STATUS
   engineering-dev-guestbook    OutOfSync     Missing
   engineering-prod-guestbook   OutOfSync     Missing

8. Delete appset

   $ kubectl delete -f appset.yaml
   applicationset.argoproj.io "guestbook" deleted
   
   $ kubectl get applicationset/guestbook -n apps
   Error from server (NotFound): applicationsets.argoproj.io "guestbook" not found
   
   $ kubectl get applications -n apps
   No resources found in apps namespace.

9. Argo CD CLI tests (use cli version >= 2.10)

# kubectl port-forward -n test service/argocd-server 8080:443
# login to argocd cli

$ argocd appset create appset.yaml 
ApplicationSet 'guestbook' created

$ argocd appset list
NAME            PROJECT  SYNCPOLICY     CONDITIONS                                                                                                                                                                                                                                           REPO                                     PATH                                                          TARGET
apps/guestbook  default  nil         [{ParametersGenerated Successfully generated parameters for all Applications 2024-02-09    05:07:05 +0530 IST True ParametersGenerated} {ResourcesUpToDate ApplicationSet up to date 2024-02-09 05:07:05 +0530 IST True    ApplicationSetUpToDate}]  https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/   {{cluster}}  HEAD
sghadi-mac:appsets sghadi$ ./argocd appset get apps/guestbook
Name:               apps/guestbook
Project:            default
Server:             {{url}}
Namespace:          guestbook
Repo:               https://github.com/argoproj/argo-cd.git
Target:             HEAD
Path:               applicationset/examples/list-generator/guestbook/{{cluster}}
SyncPolicy:         <none>

CONDITION            STATUS  MESSAGE                                                 LAST TRANSITION
ErrorOccurred        False   Successfully generated parameters for all Applications  2024-02-09 05:07:05 +0530 IST
ParametersGenerated  True    Successfully generated parameters for all Applications  2024-02-09 05:07:05 +0530 IST
ResourcesUpToDate    True    ApplicationSet up to date                               2024-02-09 05:07:05 +0530 IST

$ argocd app list
NAME                             CLUSTER                         NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                               TARGET
apps/engineering-dev-guestbook   https://kubernetes.default.svc  guestbook  default  OutOfSync  Missing  <none>      <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev   HEAD
apps/engineering-prod-guestbook  https://kubernetes.default.svc  guestbook  default  OutOfSync  Missing  <none>      <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-prod  HEAD

$ argocd app sync apps/engineering-dev-guestbook
......
GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH       HOOK  MESSAGE
       Service     dev        guestbook-ui  Synced  Healthy            service/guestbook-ui created
apps   Deployment  dev        guestbook-ui  Synced  Progressing        deployment.apps/guestbook-ui created

$ argocd app sync apps/engineering-prod-guestbook
.....
GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH       HOOK  MESSAGE
       Service     prod       guestbook-ui  Synced  Healthy            service/guestbook-ui created
apps   Deployment  prod       guestbook-ui  Synced  Progressing        deployment.apps/guestbook-ui created

$ argocd app list
NAME                             CLUSTER                         NAMESPACE  PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                     PATH                                                               TARGET
apps/engineering-dev-guestbook   https://kubernetes.default.svc  dev        default  Synced  Healthy  <none>      <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-dev   HEAD
apps/engineering-prod-guestbook  https://kubernetes.default.svc  prod       default  Synced  Healthy  <none>      <none>      https://github.com/argoproj/argo-cd.git  applicationset/examples/list-generator/guestbook/engineering-prod  HEAD

$ argocd appset delete apps/guestbook
Are you sure you want to delete 'apps/guestbook' and all its Applications? [y/n] y
applicationset 'apps/guestbook' deleted

$ argocd appset list
NAME  PROJECT  SYNCPOLICY  CONDITIONS  REPO  PATH  TARGET

$ argocd app list
NAME  CLUSTER  NAMESPACE  PROJECT  STATUS  HEALTH  SYNCPOLICY  CONDITIONS  REPO  PATH  TARGET

11. Cleanup - Disable appset in target ns

$ kubectl patch -n test argocd/argocd --type='json' -p='[{"op": "remove", "path": "/spec/applicationSet/sourceNamespaces"}]'
argocd.argoproj.io/argocd patched

$ kubectl get argocd/argocd -n test -o jsonpath='{.status.applicationSetController}'
Running

$ kubectl get rolebindings -n apps
NAME          ROLE               AGE
argocd_apps   Role/argocd_apps   8m7s

$ kubectl patch -n test argocd/argocd --type='json' -p='[{"op": "remove", "path": "/spec/sourceNamespaces"}]'
argocd.argoproj.io/argocd patched

$  kubectl get rolebindings -n apps
No resources found in apps namespace.

[jaideepr97]

Thanks a lot @svghadi !

Is a clusterrolebinding really necessary? That would allow the appset controller to exercise its privileges in all namespaces, even the ones that are not included in the whitelist
The apps-in-ay-namespaces implementation creates roles and rolebindings for each source namespace individually
(although this effort can probably be reduced by creating a single clusterrole and a rolebinding in each namespace)

Do you think we could follow that method here?

[jaideepr97]

I also wonder if we will need a separate role for the server to be able to access appsets in all these new namespaces, since the CLI interacts with the server

[svghadi]

Is a clusterrolebinding really necessary?

There is no upstream RBAC documentation for appsets in any namespace. Through trial and error, I found that the appset-controller requires atleast watch & list permissions on applications, applicationsets & secrets at cluster level to function properly. I think we could implement your suggested approach by having only necessary rules at cluster level and moving everything else to namespace-scoped.

I also wonder if we will need a separate role for the server to be able to access appsets in all these new namespaces, since the CLI interacts with the server

Good catch. I will include this.

[svghadi]

Right now the implementation expects user to enable app in any ns on the target namespace inorder for appsets in any ns to function properly. Therefore a working ArgoCD CR config of appset in any ns would like

spec:
  applicationSet:
    sourceNamespaces:
      - apps
  # appset in any ns requires apps in any ns to function properly
  sourceNamespaces:
    - apps

If we change the implementation to enable it implicitly, it would improve the UX. The ArgoCD CR config would look like

spec:
  applicationSet:
    sourceNamespaces:
      - apps

WDYT? @jaideepr97 . Should be enable app in any ns implicitly or let user set it explicitly so that they are aware of it?

[jaideepr97]

Is a clusterrolebinding really necessary?

There is no upstream RBAC documentation for appsets in any namespace. Through trial and error, I found that the appset-controller requires atleast watch & list permissions on applications, applicationsets & secrets at cluster level to function properly. I think we could implement your suggested approach by having only necessary rules at cluster level and moving everything else to namespace-scoped.

That sounds good! since the list of required permissions is quite big lets try to restrict it as much as possible

I also wonder if we will need a separate role for the server to be able to access appsets in all these new namespaces, since the CLI interacts with the server

Good catch. I will include this.

There will actually already be an existing server role for the server to access apps in that namespace, so we can probably use that to give it access to appsets in that namespace as well, instead of creating a new role just for appsets

Right now the implementation expects user to enable app in any ns on the target namespace inorder for appsets in any ns to function properly. Therefore a working ArgoCD CR config of appset in any ns would like

spec:
 applicationSet:
   sourceNamespaces:
     - apps 
# appset in any ns requires apps in any ns to function properly
 sourceNamespaces:
   - apps

If we change the implementation to enable it implicitly, it would improve the UX. The ArgoCD CR config would look like

spec:
 applicationSet:
   sourceNamespaces:
     - apps

WDYT? @jaideepr97 . Should be enable app in any ns implicitly or let user set it explicitly so that they are aware of it?

All appset source namespaces must be app source namespaces, but not all app source namespaces need to be appset source namespaces
Coupling the 2 together would make it difficult for users of apps-in-any-namespaces that are not using appsets
It would be a breaking change anyway, which we probably cannot do until we move the Argo CD CRD into its own group

[svghadi]

I didn't mean to introduce a break change. Let the users who want apps-in-any-namespaces continue using .spec.sourceNamespaces. What I am suggesting is to automatically handle enabling apps-in-any-namespaces for users who have enabled appsets-in-any-namespaces such that only adding target ns in .spec.applicationSet.sourceNamespaces is enough to enable the completely functionality.

[svghadi]

@jaideepr97 - PR is ready for a second look.

New Changes:

• Limit applicationset-controller clusterrole & clusterrolebinding permissions to watch & list instance
• Add new role & rolebinding in target namespace with required permissions for applicationset-controller
• Create/modify role for argocd-server in target namespace to grant access to appsets for CLI/UI access
• Add kuttl test

[svghadi]

Latest changes:

• Skip reconciliation of resources if appset source namespaces is not subset of app source namespaces

[jaideepr97]

LGTM @svghadi
Thanks!