SSO issue on ArgoWorkFlow after HelmChart Upgrade

[JDHawken]

Hello everyone,
I'm trying to update my ArgoWorkFlow from 3.4.8 to 3.5.8.
Everything look good but I meet issue when users (administrators included) try to loggin into ArgoWorkFlow, we got the following error :
Failed to load version/info Error: Forbidden

I looked into logs and I can find theses following messages :

time="2024-08-06T09:37:08.059Z" level=error msg="failed to perform RBAC authorization" error="no service account rule matches"
time="2024-08-06T09:37:08.059Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = not allowed"
time="2024-08-06T09:37:08.059Z" level=error msg="failed to perform RBAC authorization" error="no service account rule matches"                                  
time="2024-08-06T09:37:08.059Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = not allowed"

Here my values.yaml on this part :

  sso:
    enabled: true
    issuer: [my-domain]/auth/realms/master
    redirectUrl: [my-domain]/argo/oauth2/callback
    clientId:
      name: [my-id]
      key: [my-key]
    clientSecret:
      name: [my-secret]
      key: [my-key]
    redirectUrl: ""
    insecureSkipVerify: true
    rbac:
      enabled: true
    scopes:
      - ArgoRoles
  ssoCAcrt: [my-crt]
  ssoSecret:
    enabled: true
    name: [my-secret]
    clientId: argo
    clientSecret: [secret-in-keycloak]
  ssoAccounts:
    enabled: true
    prefix: argo
    userRule: "'ARGO_USER' in groups"
    adminRule: "'ARGO_ADMIN' in groups"

Authentication is provided by Keycloak, I'm not sure where to look...
I checked Keycloak and communication in logs and everything looking good on my part :

When I put wrong password :
13:28:21,098 WARN [org.keycloak.events] (default task-57) type=LOGIN_ERROR, realmId=master, clientId=argo, userId=9cf3f2cb-e74d-47cd-a4b0-23da81cc172b, ipAddress=, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code, redirect_uri=[my-domain]/argo/oauth2/callback, code_id=e3c5e019-64fd-420c-9e8a-5c4fa57508ed, username=[my-username], authSessionParentId=e3c5e019-64fd-420c-9e8a-5c4fa57508ed, authSessionTabId=KrTMZNxNpT0

When password is correct : No log.

I checked existence of ArgoRoles in Keycloak (Realms>Master>Configure>Client Scopes) and I can found it !
I try to back on 3.4.8 and it's worked again, I think I got problem in my values.yaml.
Any idea ?

Many thanks in advance !

[yu-croco]

Hi @JDHawken

I'm trying to update my ArgoWorkFlow from 3.4.8 to 3.5.8.

I try to back on 3.4.8 and it's worked again, I think I got problem in my values.yaml.
Any idea ?

I'm not so sure which Chart version you are using (not Argo Workflows itself), but you should check diffs in values.yaml.
e.g.) argo-workflows-0.41.13...argo-workflows-0.41.14

[yu-croco]

@JDHawken thank you for the info.
Hmmm I am not so sure the cause, it doesn't look Helm Chart causes this.
As you said earlier, your values.yaml looks ok, sso.enabled is reflected in your values.yaml.
*diff: argo-workflows-0.28.2...argo-workflows-0.41.11

Can you also ask in upstream?

*I wonder the version of k8s could be the case (some users are using too old K8s)...? 🤔

[JDHawken]

Hello,
My K8S cluster is under 1.28 ! Pretty old but I'm not sure the problem is here ^^.

I got something new !
I make comparaison between my prod and staging and find a difference...
My production server as some ServiceAccount who don't exist on my Staging Server...

I'm little bit confused because I'm not sure which ServiceAccount is used ... I didn't find any reference to "argo-webui" on my values.yaml or my deployment... I'm pretty sure the problem is here because they exist on the functionnal server and not on my other server.

To avoid the problem, I disabled sso.rbac (= false) and everything working well now !

About ServiceAccount, it's probably internal modification and I have to investigate in my side, but did you know how to specify which serviceaccount use for rbac ? How it's generated and processed ?

Thanks for trying to help me !

[yu-croco]

Hi @JDHawken , thank you for the info.

About ServiceAccount, it's probably internal modification and I have to investigate in my side, but did you know how to specify which serviceaccount use for rbac ? How it's generated and processed ?

Can you please share your values.yaml? (mainly .Values.server)
If there are some diffs in two of versions (v0.28.2 and v0.41.11), I want both of them.
*As you say, I think some of the flags maybe be wiped out or moved to other indent. 🤔

[JDHawken]

Hello,
Finally ! I found it !
It seem we got some custom templates in our ArgoWorkFlows and I deleted them when I upgraded the product, so it's not a problem with ArgoWorkFlows but in my side (I'm sorry about it, I don't have enough history to have seen it right away).

Here how I debug my deployment :

• I download on my GitHub the staging and production files (production working well, staging had the problem).
• With helm, I make two templates, one for production and one for staging

helm template --skip-tests -f values-common.yaml -f values-prod.yaml -n [my-namespace] --kube-context [my-context] [my-release] . >prod.yaml
helm template --skip-tests -f values-common.yaml -f values-staging.yaml -n [my-namespace] --kube-context [my-context] [my-release] . >staging.yaml

• Now, I make diff between prod & staging and I can see that :
  

I retrieved the missing files and uploaded them to my Staging server, now everything is okay and I can validate it !

Again, many thanks @yu-croco for the help !
(Despite the fact that the problem was not with Argo but with me 🤣 )