Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

[dgoldssfo]

Note - I have this problem with 1.18.1 and 1.19.0
Is there a version I can back off to that will work?

Steps to reproduce the issue:

1. forgeops git:(master) ✗ minikube start --memory=10240 --cpus=3 --disk-size=40g --cni=true --vm=true
   --driver=virtualbox --bootstrapper kubeadm --kubernetes-version=1.20.0
2. minikube addons enable ingress

Full output of minikube start command used, if not already included: (also includes o/p from minikube addons enable ingress)

➜ forgeops git:(master) ✗ minikube start --memory=10240 --cpus=3 --disk-size=40g --cni=true --vm=true
--driver=virtualbox --bootstrapper kubeadm --kubernetes-version=1.20.0
😄 minikube v1.19.0 on Darwin 11.2.2
✨ Using the virtualbox driver based on user configuration
💿 Downloading VM boot image ...
> minikube-v1.19.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
> minikube-v1.19.0.iso: 244.49 MiB / 244.49 MiB 100.00% 5.94 MiB p/s 41.34
👍 Starting control plane node minikube in cluster minikube
💾 Downloading Kubernetes v1.20.0 preload ...
> preloaded-images-k8s-v10-v1...: 491.68 MiB / 491.68 MiB 100.00% 5.63 MiB
🔥 Creating virtualbox VM (CPUs=3, Memory=10240MB, Disk=40960MB) ...
🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.4 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
➜ forgeops git:(master) ✗ minikube addons enable ingress
▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
🔎 Verifying ingress addon...

❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

I tried to pull the Docker file that was causing the problem; stuck on waiting:

➜ forgeops git:(master) ✗ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-dhp4x 0/1 Completed 0 86m
ingress-nginx ingress-nginx-admission-patch-h74ws 0/1 Completed 0 86m
ingress-nginx ingress-nginx-controller-5d88495688-2bgln 0/1 ImagePullBackOff 0 86m
kube-system coredns-74ff55c5b-lsmmx 1/1 Running 0 87m
kube-system etcd-minikube 1/1 Running 0 87m
kube-system kindnet-cq5xj 1/1 Running 0 87m
kube-system kube-apiserver-minikube 1/1 Running 0 87m
kube-system kube-controller-manager-minikube 1/1 Running 0 87m
kube-system kube-proxy-zrscz 1/1 Running 0 87m
kube-system kube-scheduler-minikube 1/1 Running 0 87m
kube-system storage-provisioner 1/1 Running 0 87m
➜ forgeops git:(master) ✗ kubectl describe pod/ingress-nginx-controller-5d88495688-2bgln --namespace=kube-system
Error from server (NotFound): pods "ingress-nginx-controller-5d88495688-2bgln" not found
➜ forgeops git:(master) ✗ kubectl describe pod/ingress-nginx-controller-5d88495688-2bgln --namespace=ingress-nginx
Name: ingress-nginx-controller-5d88495688-2bgln
Namespace: ingress-nginx
Priority: 0
Node: minikube/192.168.99.100
Start Time: Mon, 19 Apr 2021 11:58:25 -0700
Labels: addonmanager.kubernetes.io/mode=Reconcile
app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
gcp-auth-skip-secret=true
pod-template-hash=5d88495688
Annotations:
Status: Pending
IP: 10.244.0.5
IPs:
IP: 10.244.0.5
Controlled By: ReplicaSet/ingress-nginx-controller-5d88495688
Containers:
controller:
Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP
Args:
/nginx-ingress-controller
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--report-node-internal-ip-address
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-5d88495688-2bgln (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-qzbqr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
ingress-nginx-token-qzbqr:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-token-qzbqr
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning Failed 41m (x11 over 84m) kubelet Error: ErrImagePull
Normal Pulling 17m (x15 over 87m) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a"
Warning Failed 7m6s (x263 over 84m) kubelet Error: ImagePullBackOff
Normal BackOff 2m10s (x280 over 84m) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a"
➜ forgeops git:(master) ✗ minikube ssh
_ _
_ _ ( ) ( )
___ ___ () ___ ()| |/') _ _ | |_ __
/' _ _| |/' _ \| || , < ( ) ( )| '_\ /'\ | ( ) ( ) || || ( ) || || |\\ | () || |) )( /
() () ()()() ()()() ()\___/'(_,__/'__)

$ docker pull k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a
k8s.gcr.io/ingress-nginx/controller@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a: Pulling from ingress-nginx/controller
596ba82af5aa: Already exists
b198ba082145: Already exists
c7166c196d5e: Already exists
addc1fbd9812: Already exists
f9949b841da7: Already exists
97335954f799: Waiting
826bbedb287b: Waiting

[dgoldssfo]

Stuck on "Waiting" for at least ten minutes now.

[dgoldssfo]

A friend suggested I blow away my ~/.minikube directory and try it, which solved the problem.

If you don't want to investigate why having cruft in that directory caused this problem, go ahead and close this. But it might be good to look into it. Your call, thanks for all the great work you folks do.

[spowelljr]

@dgoldssfo Thanks for the info, did you update from an older version of minikube previous to this?

[dgoldssfo]

Yes, I updated from 1.18.1 to 1.19.0.

But note - enabling the ingress addon had gotten very slow (like, it took several minutes). After deleting .minikube, it was back to taking < 10 seconds.

[spowelljr]

Thank you for you info, you're not the only person who experienced this issue. I'll see if I can replicate the same issue.

[dgoldssfo]

I'm also having other ingress issues. Investigating now.

[dgoldssfo]

Could be our app, but maybe this'll help:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

[dgoldssfo]

Looks like it's the ingress controller plugin. Is there a new one in 1.19?

Trying to ctreate this ingress fails, for example:

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: default-issuer
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite ^(/platform)$ $1/ redirect;
nginx.ingress.kubernetes.io/rewrite-target: /$1
labels:
app.kubernetes.io/part-of: forgerock
name: platform-ui
spec:
rules:

• host: default.iam.example.com
  http:
  paths:
  • backend:
    service:
    name: admin-ui
    port:
    number: 8080
    path: /platform/?(.*)
    pathType: Prefix
    tls:
• hosts:
  • default.iam.example.com
    secretName: sslcert

[medyagh]

thank you everyone for reporting this bug, this seems to be side effect of this PR #10879 that we tried to update the ingress to new version, unfortunately our integration test did not catch this problem,
Please stay tuned, we will create a fix for this and hopefully by adding an integration test this should never happen again.

meanwhile could you try

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

and see if that fixes the problem ?

I suspect this is dupe of #11121

[vgryzunov]

Tried kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission. Did not help.
Had to re-create minikube by

minikube delete --all --purge
minikube addons enable ingress

[spowelljr]

Hi @dgoldssfo, thanks for reporting your issue!

This is a duplicate of #10544, in order to keep all discussion of this issue centralized I'm going to close this issue.

Please follow along with #10544 and contribute any findings there, thanks for your understanding!

[gilkremer123]

Had the same issue (on windows). could not enable ingress. turns out running CMD as an administrator solved it :)
So it was just a permissions issue

[Monte-Christo]

Had the same issue on Win 10 Enterprise. When I switched from Admin PowerShell to Admin Cmd Shell, the command completed without any fuss. Go figure.

[Richard-code-gig]

In Admin CMD this should resolve it:
minikube delete --all --purge
minikube start
minikube addons enable ingress