Multi-namespace workflows

[naor2013]

Summary

How do you suggest handling multiple namespaces that run Argo workflows in the new version?
The ways I’m thinking about right now are either:
a) workflow controller for each namespace and an Argo server for each namespace.
b) workflow controller for each namespace and a “managing” namespace which has a Argo server for each namespace with the “managed-namespace” patameter.

Is there a better way? Do you think it will be a good idea for you to implement an option to have one Argo server with multiple managed-namespaces for it?

Motivation

We are the MLOps team and we want every DS team to have their own namespace with their own resource limitations and only show in the Argo UI their own workflows.

[maryoush]

Hi
what with the case for multi tenant spanned workflow ?
#2042

[luozhaoyu]

I have similar issue: different team may want to create pods in different namespaces - their individual namespace would have different pods to interact with.
Currently --managed-namespace is a single string: https://github.com/argoproj/argo/blob/69179e72c0872cde9131cc9d68192d5c472d64c9/cmd/argo/commands/server.go#L35

[alexec]

See #3523

[dudicoco]

I was able to achieve this by using the following workaround:

1. Run workflow-controller and argo-server without the --namespaced flag
2. Create a cluster role for workflow-controller with read only permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-workflows-controller
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - argoproj.io
  resources:
  - workflowtemplates
  - workflowtemplates/finalizers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - argoproj.io
  resources:
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - get
  - list
  - watch

3. Bind the cluster role to the workflow-controller service account.
4. Provide write access to specific namespaces by creating a role with the appropriate permissions in that namespace and binding it to the workflow-controller service account.

[syu-lk4b]

I tried the above steps, been able to create the job in a different namesapce, however the job stucks in pending status forever, any idea

argo list                                                       
NAME                STATUS    AGE   DURATION   PRIORITY
sumc-cct-job7s7t9   Pending   15m   0s         0
sumc-cct-job9mphx   Pending   17m   0s         0
sumc-cct-jobbg7xr   Pending   19m   0s         0
sumc-cct-job29btn   Pending   26m   0s         0
sumc-cct-jobzlgpl   Pending   28m   0s         0
sumc-cct-jobdg2np   Pending   28m   0s         0
sumc-cct-jobjvh57   Pending   28m   0s         0
sumc-cct-jobbm6w7   Pending   33m   0s         0
sumc-cct-job9292s   Pending   49m   0s         0
sumc-cct-jobmvpkk   Pending   1h    0s         0

[alexec]

You may now test using #6587

[tachyus-ryan]

@dudicoco could you elaborate a bit more? I am trying to set this up, but the workflows are not starting. I think I also need to replace, not just add, the cluster role you suggest above.

[dudicoco]

@tachyus-ryan we are no longer using argo-workflows after our initial POC so I don't really remember the entire configuration.

However, if you post your yaml manifests here perhaps I could help you troubleshoot the problem.

[alexec]

Updated version for testing:

https://github.com/argoproj/argo-workflows/releases/tag/v0.0.0-dev-mc-8

[agilgur5]

To make multi-namespace Workflows properly support the k8s security model, we'd probably need a ClusterWorkflow CR. A Workflow CR does not suffice as it is namespace-scoped.
Although you can certainly workaround this by using the Workflows of Workflows pattern with sufficient RBAC to create workflows in other namespaces.