[Minikube] webhook-server-tls secret not created

[jednymslowem]

What steps did you take:

webhook-server-tls secret not created during Kubeflow Pipelines standalone deployment

What happened:

Cache server got stuck in ContainerCreating state, because of the missing secret

Environment:

Mac OS, I use minikube cluster bundled with Docker for desktop

How did you deploy Kubeflow Pipelines (KFP)?

export PIPELINE_VERSION=0.5.1
kubectl apply -k github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION

KFP version: 0.5.1

KFP SDK version: not installed

Anything else you would like to add:

It happens after a second deployment (when I deploy for the first time on a fresh cluster it works fine, then when I delete the deployment with:

export PIPELINE_VERSION=0.5.1
kubectl delete -k github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION
kubectl delete -k github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION

and recreate it with the command above, the secret is not being created.

When I run ./deploy-cache-service.sh manually, the secret is created and the cache server is starting as expected.

/kind bug
/area backend

[Ark-kun]

Can you please check the logs of cache-deployer?

P.S. Are you able to run any pipelines?

[tarjintor]

I got the same issue,my logs are

ubuntu@t224:~$ kubectl -n kubeflow logs -f cache-deployer-deployment-5f94848fd9-xbjws --tail=100
+ echo 'Start deploying cache service to existing cluster:'
+ NAMESPACE=kubeflow
+ MUTATING_WEBHOOK_CONFIGURATION_NAME=cache-webhook-kubeflow
+ kubectl get mutatingwebhookconfigurations cache-webhook-kubeflow --namespace kubeflow --ignore-not-found
Start deploying cache service to existing cluster:
cache-webhook-kubeflow   2020-05-25T16:23:28Z
+ grep cache-webhook-kubeflow -w
+ echo 'Webhook is already installed. Sleeping forever.'
+ sleep infinity
Webhook is already installed. Sleeping forever.

My k8s cluster already have a prometheus server before I use pipelines
And it's a regular k8s cluster,not minikube

But I can run toturrial pipelines
So I don't need cache server in fact?

[Ark-kun]

I got the same issue,my logs are

There should be more logs since some service had successfully deployed cache-webhook-kubeflow.

Can you please check the previous pods of that deployment?

If that does not work, you can try deleting the cache-webhook-kubeflow config object and then restarting the cache-deployer-deployment-5f94848fd9-xbjws pod.

[Ark-kun]

So I don't need cache server in fact?

The caching server works as a standalone service. If it's deployed correctly, it will make your pipelines run much faster by skipping already completed steps. If it does not work, your pipelines just run as usual without skipping any steps.

[tarjintor]

I delete cache-webhook-kubeflow and it works,thanks

[Ark-kun]

If this happens again try to collect the cache deployer logs across all restarts, so we can see how it was creating the secret.

[jednymslowem]

I think I got to the bottom of it. cache-webhook-kubeflow is created all right, but it is not being deleted by kubectl delete -k github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION . So when I redeploy Kubeflow Pipelines to the same cluster again, the old cache-webhook-kubeflow is still there and it is leading to the logs @tarjintor pasted above.

Adding kubectl delete mutatingwebhookconfigurations cache-webhook-kubeflow to my deletion script solved the issue.

[Ark-kun]

So when I redeploy Kubeflow Pipelines to the same cluster again, the old cache-webhook-kubeflow is still there

I wonder why the TLS secret was deleted. Was the namespace deleted?