RBAC: access denied when making a request to ISVC through the local gateway with istio sidecar enabled

[NohaIhab]

Bug Description

ISVCs are not reachable through the local gateway with istio sidecar enabled (the default in kubeflow user namespaces), where a 403 response is received with RBAC: access denied.

To Reproduce

1. deploy ckf 1.8/stable and configure dashboard access

2. create a kubeflow user namespace by logging into the dashboard

3. create an isvc in the kubeflow user namespace:

kubectl apply -n admin -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

4. create a temp pod to test requests to the isvc

kubectl run netshoot --rm -i --tty --image nicolaka/netshoot -- /bin/bash

alternatively, you can do the same from a notebook pod

5. from inside the pod, make a prediction request to the isvc using the ISVC's status.address.url

# prepare inference input
cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

# curl isvc
curl -v -H "Content-Type: application/json" http://sklearn-iris.admin.svc.cluster.local/v1/models/sklearn-iris:predict -d @./iris-input.json

Environment

kubeflow 1.8/stable
microk8s 1.25-strict/stable
juju 3.1/stable

Relevant Log Output

# prepare inference input
cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

# curl the isvc with the input
curl -v -H "Content-Type: application/json" http://sklearn-iris.admin.svc.cluster.local/v1/models/sklearn-iris:predict -d @./iris-input.json
* Host sklearn-iris.admin.svc.cluster.local:80 was resolved.
* IPv6: (none)
* IPv4: 10.152.183.48
*   Trying 10.152.183.48:80...
* Connected to sklearn-iris.admin.svc.cluster.local (10.152.183.48) port 80
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.admin.svc.cluster.local
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 76
> 
< HTTP/1.1 403 Forbidden
< content-length: 19
< content-type: text/plain
< date: Tue, 27 Feb 2024 13:38:24 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 25
< 
* Connection #0 to host sklearn-iris.admin.svc.cluster.local left intact
RBAC: access denied

### Additional Context

_No response_

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5413.

This message was autogenerated