Downloading an artifact RBAC: access denied

[NohaIhab]

This bug was faced in the pre-release testing of 1.8

Bug Description

After upgrading the kubeflow-profiles image to 1.8.0-rc.2 in PR canonical/kubeflow-profiles-operator#155, one cannot download an artifact from a step in a Pipeline run, due to the AuthorizationPolicy applied in the profile's namespaces being:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    role: admin
    user: admin
  creationTimestamp: "2023-10-20T11:36:24Z"
  generation: 1
  name: ns-owner-access-istio
  namespace: admin
  ownerReferences:
  - apiVersion: kubeflow.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Profile
    name: admin
    uid: 4ac4ba90-d64f-470b-8562-d965b4dee3f1
  resourceVersion: "70076"
  uid: 76fde5db-f822-40ac-bb4d-68db474e17e3
spec:
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/kubeflow/sa/istio-ingressgateway-workload-service-account
        - cluster.local/ns/kubeflow/sa/kfp-ui
    when:
    - key: request.headers[kubeflow-userid]
      values:
      - admin
  - when:
    - key: source.namespace
      values:
      - admin
  - to:
    - operation:
        paths:
        - /healthz
        - /metrics
        - /wait-for-drain
  - from:
    - source:
        principals:
        - cluster.local/ns/kubeflow/sa/jupyter-controller
    to:
    - operation:
        methods:
        - GET
        paths:
        - '*/api/kernels'

To Reproduce

1. Deploy kubeflow latest/edge
2. juju refresh kubeflow-profiles --channel=latest/edge/pr-155 --resource profile-image=docker.io/kubeflownotebookswg/profile-controller:v1.8.0-rc.2 --resource kfam-image=docker.io/kubeflownotebookswg/kfam:v1.8.0-rc.2

Environment

juju 3.1/stable
microk8s 1.25-strict/stable

Relevant log output

RBAC: access denied

Additional context

No response

[NohaIhab]

the issue here is that the kfp pod does not have an istio sidecar, so it is not using mTLS authentication. mTLS is needed because the AuthorizationPolicy specifies a source.principal, from the istio docs:

This field requires mTLS enabled

[NohaIhab]

To fix this temporarily and unblock the release, we can apply an AuthorizationPolicy with a more relaxed rule, so that it allows traffic from the pod without authentication. This fix would introduce a security issue, so it must be tracked and addressed in the next cycle.

[NohaIhab]

Elaborating on the above, what blocks us from downloading an artifact from the Pipelines UI is specifically this rule in the AuthorizationPolicy:

  - from:
    - source:
        principals:
        - cluster.local/ns/kubeflow/sa/istio-ingressgateway-workload-service-account
        - cluster.local/ns/kubeflow/sa/kfp-ui
    when:
    - key: request.headers[kubeflow-userid]
      values:
      - admin

to fix this, we need a rule that doesn't check on the source of the request, so the new rule should be:

  - when:
    - key: request.headers[kubeflow-userid]
      values:
      - '*'

the value of the kubeflow-userid header is set to '*' in order to match kubeflow admin AND contributor users in the case that contributors are added.

[kimwnasptd]

Some more context on this. The upstream design is described in kubeflow/pipelines#3554

Use Cases

• Artifacts are now linked to namespaces (via query parameter)
• The AuthorizationPolicy only will control which users can hit the ml-pipeline-ui-artifact service, and thus download the "namespaced" artifacts
• To be fully secure, each namespace should have different MinIO credentials
  • If all ml-pipeline-ui-artifacts have same credentials, users can still download everything
  • ml-pipeline-ui-artifacts is a crown jewel, that needs to be protected, as it contains creds to download artifacts in this namespace

Upstream Design

• The ml-pipeline-ui redirects requests to ml-pipeline-ui-artifact in user ns
• There is a namespace=... query parameter, used by ml-pipeline-ui
• The ml-pipeline-ui will add kubeflow-userid header
• The request to user namespace will be subject to AuthorizationPolicy
  • This time the source principal is the ml-pipeline-ui

Flow

1. Browser sends request to /pipeline/artifacts/minio/mlpipeline/v2/artifacts...
2. The VirtualService for /pipeline redirects everything to ml-pipeline-ui
3. ml-pipeline-ui has ARTIFACTS_SERVICE_PROXY_ENABLED: "true"
4. It will redirect the request to ml-pipeline-ui-artifact svc in user namespace

Note that all the above are referring to upstream Kubeflow. Some names will be different for Charmed Kubeflow.