[K3S] Can't start cache-deployer-deployment

[haibingzhao]

What steps did you take:

• Downloaded https://github.com/kubeflow/pipelines/releases/tag/1.0.0-rc.2
• removed the gcp/inverse-proxy

# - ../gcp/inverse-proxy

• Deployed the pipeline locally with K3D:

kubectl apply -k manifests/kustomize/cluster-scoped-resources
kubectl apply -k manifests/kustomize/env/dev/

• k3d and k3s version is:

➜  ~ k3d --version
k3d version v3.0.0-rc.5
k3s version v1.18.4-k3s1 (default)

What happened:

The module cache-deployer-deployment can't start because of:

+ kubectl certificate approve cache-server.kubeflow
No resources found
Error from server (Forbidden): certificatesigningrequests.certificates.k8s.io "cache-server.kubeflow" is forbidden: user not permitted to approve requests with signerName "kubernetes.io/legacy-unknown"

What did you expect to happen:

Environment:

Localhost environment on K3D:

k3d create kubeflow -w 3
k3d get kubeconfig kubeflow --switch

How did you deploy Kubeflow Pipelines (KFP)?

kubectl apply -k manifests/kustomize/cluster-scoped-resources
kubectl apply -k manifests/kustomize/env/dev/

KFP version: 1.0.0-rc.2

KFP SDK version: not use

Anything else you would like to add:

Related to this recent change in Kube: https://github.com/kubernetes/kubernetes/pull/86933/files
I think the cache-deployer clusterrole needs to be tweaked（manifests/kustomize/base/cache-deployer/cluster-scoped/cache-deployer-clusterrole.yaml）:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: kubeflow-pipelines-cache-deployer-clusterrole
  name: kubeflow-pipelines-cache-deployer-clusterrole
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  - certificatesigningrequests/approval
  verbs:
  - create
  - delete
  - get
  - update
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  verbs:
  - create
  - get
- apiGroups:
  - certificates.k8s.io
  resources:
  - signers
  resourceNames:
  - kubernetes.io/legacy-unknown
  verbs:
  - approve

add 'approve' (verb) 'signers' (resource) for 'certificates.k8s.io' (apiGroup), it works now. report this bug.

/kind bug
/area backend

[Bobgy]

/assign @Ark-kun
@haibingzhao Can you explain what kubernetes version you are using?

[Bobgy]

Ohhh, is it k3s version v1.18.4-k3s1 (default)?

[haibingzhao]

Ohhh, is it k3s version v1.18.4-k3s1 (default)?

yes

[Bobgy]

I think we have only been testing on Kubernetes 1.14 which is the stable channel in GCP.

PR welcomed in making it compatible with 1.18

[alfsuse]

Hi, @Bobgy I see this issue is still open, I've tested the changes merged in #4246 on k3s and on Kind with K8s 1.18, I've also tested it on a "standard" K8s (not DinD) and I can confirm it works. Do you need still/more help on this? If so how can I help you?

[Bobgy]

Thank you!
/close

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

Thank you!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ReggieCarey]

Confirmed that the bug still exists in 1.2 but the above repair is valid on Kubernetes 1.18.9.