Ambassador Addon fails to start

[tbcashy]

Steps to reproduce the issue:

1. (minikube delete)
2. minikube start
3. minikube addons enable ambassador
4. Wait :P
5. minikube kubectl -- get pods --namespace ambassador
6. (minikube kubectl -- logs ambassador-7484d7d786-f2rf5 --namespace ambassador)

Enabling the Ambassador-Addon does work, however the 3 Pods (not the ambassador-operator) won´t start properly. Instead they log the following message over and over:

Pod-Log:
E0105 16:48:22.689662 1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

It seems as if the problem is caused by the new Ambassador-Version (1.10.0). In Version 1.9.1 everything works fine.

Full Pod-Log:

2021/01/05 16:47:48 Started Ambassador
2021/01/05 16:47:49 AMBASSADOR_CLUSTER_ID=40e97ad7-c84a-5e4d-aad1-e4ad25ca2034
time="2021-01-05 16:47:49" level=info msg="[pid:21] started command []string{\"diagd\", \"/ambassador/snapshots\", \"/ambassador/bootstrap-ads.json\", \"/ambassador/envoy/envoy.json\", \"--notices\", \"/ambassador/notices.json\", \"--port\", \"8004\", \"--kick\", \"kill -HUP 1\"}" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:143" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stdin  < not logging input read from file /dev/stdin" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:145" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stdout > not logging output written to file /dev/stdout" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:148" CMD=entrypoint PID=1 THREAD=/diagd
time="2021-01-05 16:47:49" level=info msg="[pid:21] stderr > not logging output written to file /dev/stderr" func="github.com/datawire/dlib/dexec.(*Cmd).Start" file="github.com/datawire/dlib@v1.1.0/dexec/cmd.go:151" CMD=entrypoint PID=1 THREAD=/diagd
E0105 16:47:49.373264       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: AMBASSADOR_FAST_RECONFIGURE disabled, not initializing cache
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: WILL NOT update Mapping status
2021-01-05 16:47:50 diagd 1.10.0 [P21TMainThread] INFO: thread count 25, listening on 0.0.0.0:8004
E0105 16:47:50.969162       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021-01-05 16:47:51 diagd 1.10.0 [P21TMainThread] INFO: Ambassador 1.10.0 booted
[2021-01-05 16:47:51 +0000] [21] [INFO] Starting gunicorn 20.0.4
[2021-01-05 16:47:51 +0000] [21] [INFO] Listening at: http://0.0.0.0:8004 (21)
[2021-01-05 16:47:51 +0000] [21] [INFO] Using worker: threads
[2021-01-05 16:47:51 +0000] [27] [INFO] Booting worker with pid: 27
2021-01-05 16:47:51 diagd 1.10.0 [P27TAEW] INFO: starting Scout checker and timer logger
2021-01-05 16:47:51 diagd 1.10.0 [P27TAEW] INFO: starting event watcher
E0105 16:47:53.660978       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:47:57.563608       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
2021/01/05 16:47:59 Memory Usage 0.07Gi (12%)
    PID 1, 0.06Gi: busyambassador entrypoint 
    PID 21, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
    PID 27, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 
E0105 16:48:05.266506       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:48:22.689662       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:49:02.902865       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0105 16:49:58.781330       1 reflector.go:178] k8s.io/client-go@v0.18.4/tools/cache/reflector.go:125: Failed to list *unstructured.Unstructured: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ambassador:ambassador" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

Running Pods:

NAME                                  READY   STATUS    RESTARTS   AGE
ambassador-7484d7d786-f2rf5           0/1     Running   0          9m20s
ambassador-7484d7d786-f6648           0/1     Running   0          9m20s
ambassador-7484d7d786-mpg4h           0/1     Running   0          9m20s
ambassador-operator-fcb94f945-kckgx   1/1     Running   0          9m45s  

Addons which are also enabled:

• storage-provisioner
• default-storageclass

Specs:
Driver: Parallels
CPUs: 12
RAM: 12GB
Disk: 30GB
OS: macOS 11.1

Confirmend the error also exists with the docker driver.

[sharifelgamal]

This seems to be a configuration issue with our ambassador addon yaml. This addon was contributed by the community, so the minikube maintainers can't guarantee being able to maintain these properly. We will happily review any PRs to fix this though.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[sharifelgamal]

Freezing this so it doesn't accidentally get closed. Help still wanted!

[kadern0]

@sharifelgamal, it seems Ambassador has been renamed to "emissary-ingress" https://www.cncf.io/blog/2021/04/13/emissary-ingress-formerly-ambassador-is-now-a-cncf-incubating-project/ so maybe this issue needs to be redefined.