[v1] jettech/kube-webhook-certgen is not compatible with 1.22+ #7418
Comments
|
@maybe-sybr: This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
needs-triage
|
I encountered this issue through the 4.0.0 chart (beta) that was just released, I think I was able to resolve it by disabling the The Kubernetes cluster is 1.22.0-rc.0 |
|
I actually hit some further issues late yesterday with the Applying a templatised YAML for one of my charts: $ kubectl apply -f my-app.yaml
service/my-app created
deployment.apps/my-app created
pod/my-app-test-connection created
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\",\"nginx.ingress.kubernetes.io/rewrite-target\":\"/$2\",\"nginx.ingress.kubernetes.io/x-forwarded-prefix\":\"/my-app\"},\"labels\":{\"app.kubernetes.io/instance\":\"my-app\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"my-app\",\"app.kubernetes.io/version\":\"1.6.0\",\"helm.sh/chart\":\"my-app-1.2.0\"},\"name\":\"my-app\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"frontend.app.lan\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"my-app\",\"port\":{\"number\":80}}},\"path\":\"/my-app(/|$)(.*)\",\"pathType\":\"Prefix\"}]}}]}}\n"}},"spec":{"rules":[{"host":"frontend.app.lan","http":{"paths":[{"backend":{"service":{"name":"my-app","port":{"number":80}}},"path":"/my-app(/|$)(.*)","pathType":"Prefix"}]}}]}}
to:
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind:
"networking.k8s.io/v1, Kind=Ingress"
Name: "my-app", Namespace: "default"
for: "my-app.yaml": Internal error occurred: failed calling webhook
"validate.nginx.ingress.kubernetes.io": Post
"https://ingress-nginx-controller-admission.kube-system.svc:443/networking/v1/ingresses?timeout=30s":
x509: certificate signed by unknown authorityThat kind of makes sense I guess, since it's likely that without the patching admission webhook, there might be some stubbed TLS certificate rather than the one which should have been minted. I may have to run through this stuff again to see if I've made some mistake. |
|
That's what I'm seeing as well. At first glance it looked like disabling the patch hook solved the problem, but it only stopped the ingress controller from crashing. I got the same x509 errors you saw after that, you're not doing anything wrong (or we both are!) |
|
In my case ValidatingWebhookConfiguration stayed installed from previous helm installation. I deleted mine with and install 4.0.0 helm chart with |
|
I can confirm, the 1.0.0-beta.0 image works for me with the 4.0.0 chart, as long as I disable admission webhooks globally. Disabling the patch hook by itself was not enough. I think the other hook must depend on the certificates that are applied through the patch hook. I'm sure this will have to be resolved somehow before the 1.0.0 final can be released. But I am currently using Kubernetes 1.22.0, Cert-manager 1.5.0-beta.0, and ingress-nginx together (and it is glorious 🎉) thanks for the tip @fracarvic |
|
/priority critical-urgent |
k8s-ci-robot
added
priority/critical-urgent
|
/cc
yep. we need to discuss a solution.
|
|
I've reached @vsliouniaev on Slack before thinking about fork or other solutions. I'm ready to submit a PR, just checking if it works as expected here :) |
|
Folks, can you please test the webhook with image: rpkatz/kube-webhook-certgen:v1.5.2 And check if the problem persists? If this is solved, I'm going to submit a PR to the original project |
|
fyi patch works: Gonna open the PR here |
|
That Thanks for chasing this up! |
|
Great news. Thank you all for sticking with us :) I’m expecting some answer on this by tuesday this week (aka tomorrow in my timezone) and will discuss with other maintainers the best approach so this wont be a showstopper. Will keep this issue open right now, as there’s no official solution yet |
|
Tested helm chart 4.0.0-beta.1 with webhooks enabled and works perfectly, can create new ingress resources without problem. Thanks. |
|
Just tested helm chart 4.0.0-beta.2 with all of my customizations removed, with the included image and with re-enabled admission webhooks. It's working for me. 👍 |
|
hello @maybe-sybr @kingdonb @fracarvic , We have released a new beta This contains a new image for the certgen k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068 We are hoping you can test this beta.2 and provide feedback. |
|
Tested helm chart 4.0.0-beta.2 with default admission configuration from chart and all is working well. |
|
Thanks for updating @kingdonb @fracarvic |
|
@longwuyuan - the new chart also works for me with all of my changes reverted and replaced with a simple Edit: Turns out I tested on a 1.21+ cluster since I had to walk it back for other reasons. In any case, at least the fact that the new chart and images worked suggests that they're using the ingress/v1 API happily now, and that the new certgen image was pulled as expected. |
|
Thank you for updating @maybe-sybr |
|
/close |
|
@longwuyuan: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Happy to report success on k8s v1.21.0 as well. Thanks to the team for your hard work getting this quickly resolved! |
|
@briantopping thank you |
|
Out of curiosity... What is the command you are using to install the "specific version"? helm search repo ingress-nginx
NAME CHART VERSION APP VERSION DESCRIPTION
ingress-nginx/ingress-nginx 3.35.0 0.48.1 Ingress controller for Kubernetes using NGINX a...helm search repo ingress-nginx --devel
NAME CHART VERSION APP VERSION DESCRIPTION
ingress-nginx/ingress-nginx 4.0.0-beta.3 1.0.0-beta.3 Ingress controller for Kubernetes using NGINX a...I am trying helm fetch --untar ingress-nginx/ingress-nginx --version 4.0.0-beta.3But the Changelog is still 3.34. I do note a continued issues around "--version" over helm/helm#8739. Even when I do fetch without untar, the tar image references the old Changelog and more importantly the tag is 1.5.1 not 1.5.2. I suspect I am missing something very obvious :). |
|
--devel
Thanks,
; Long
…On Sat, 14 Aug, 2021, 11:44 PM Chris, ***@***.***> wrote:
Out of curiosity...
What is the command you are using to install the "specific version"?
helm search repo ingress-nginx
NAME CHART VERSION APP VERSION DESCRIPTION
ingress-nginx/ingress-nginx 3.35.0 0.48.1 Ingress controller for Kubernetes using NGINX a...
helm search repo ingress-nginx --devel
NAME CHART VERSION APP VERSION DESCRIPTION
ingress-nginx/ingress-nginx 4.0.0-beta.3 1.0.0-beta.3 Ingress controller for Kubernetes using NGINX a...
I am trying
helm fetch --untar ingress-nginx/ingress-nginx --version 4.0.0-beta.3
But the Changelog is still 3.34.
I do note a continued issues around "--version" over helm/helm#8739
<helm/helm#8739>.
But I cant ignore people in this thread saying it worked.
Even when I do fetch without untar, the tar image references the old
Changelog and more importantly the tag is 1.5.1 not 1.5.2.
I suspect I am missing something very obvious :).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#7418 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABGZVWXO3DTUIW2IYKVWWETT42XA3ANCNFSM5BN54PVA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
The chart's changelog doesn't appear to have been updated yet. Check the |
|
Can you check the release we made 2 days ago and update ; |
|
@maybe-sybr thank you. This worked. A nice alternative to "fetch --untar". The error
How I got here rm -rf output/ingress-nginx/
helm template frontdoor ingress-nginx/ingress-nginx -f custom/ingress-nginx.yaml --output-dir ./output
kubectl apply -f output/ingress-nginx/templatesMore output 112s Normal Started pod/svclb-frontdoor-ingress-nginx-controller-xxfnp Started container lb-port-443
112s Normal Started pod/svclb-frontdoor-ingress-nginx-controller-xtr9x Started container lb-port-443
101s Warning FailedMount pod/frontdoor-ingress-nginx-controller-5b59c9bb8c-pc4mr Unable to attach or mount volumes: unmounted volumes=[webhook-cert kube-api-access-bm55g], unattached volumes=[webhook-cert kube-api-access-bm55g]: timed out waiting for the condition
49s Warning FailedMount pod/frontdoor-ingress-nginx-controller-5b59c9bb8c-stwzq MountVolume.SetUp failed for volume "webhook-cert" : secret "frontdoor-ingress-nginx-admission" not foundvalues filecontroller:
config:
log-format-upstream:
'{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",
"x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user":
"$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":$status,
"vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query":
"$args", "request_length": $request_length, "duration": $request_time,"method":
"$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent"
}'The issue still seems to be linked to:
It works when I disable the adminssionWebhooks. admissionWebhooks:
enabled: falseIt does not work if I use: admissionWebhooks:
patch:
image:
image: rpkatz/kube-webhook-certgen
tag: v1.5.2I need to understand why I would want admissionWebhooks enabled. But I am happy to get it working with it globally off. Not sure if it helps, but if you wondered, I am running k3s cluster. kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3+k3s1", GitCommit:"1d1f220fbee9cdeb5416b76b707dde8c231121f2", GitTreeState:"clean", BuildDate:"2021-07-22T20:52:14Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} |
NGINX Ingress controller version: v1.0.0-beta.1
Kubernetes version (use
kubectl version): 1.22+ server (usernetes v20210708.0)Environment: Bare metal usernetes
uname -a): 5.13.6-200.fc34.x86_64What happened:
The 1.0.0-beta.1 chart and
baremetal/deploy.yamlusejettech/kube-webhook-certgen:v1.5.1as an admission hook to patch in certs. This image attempts to use admissionregistration/v1beta1 which disappeared in API 1.22. The main repository has an outstanding issue (jet/kube-webhook-certgen#30) to update to using v1 of this API but it hasn't been worked on AFAICT.This manifests in the following error when attempting to set up an ingress-nginx on a 1.22+ server with the default chart values or example manifest YAMLs:
What you expected to happen:
How to reproduce it:
Run a 1.22+ server
I used usernetes v20210708.0 - the latest release from https://github.com/rootless-containers/usernetes/releases/latest
Install ingress-nginx with helm using a values file to set the v1.0.0-beta.1 tag and hash
sh-5.1$ cat ingress-nginx.values.yaml controller: image: tag: "v1.0.0-beta.1" digest: "sha256:f058f3fdc940095957695829745956c6acddcaef839907360965e27fd3348e2e" sh-5.1$ helm install test-ingress ./charts/ingress-nginx/ --values ingress-nginx.values.yaml ^C # Waited for a while here, it gets stuck sh-5.1$ kubectl get all -A NAMESPACE NAME READY STATUS RESTARTS AGE default pod/test-ingress-ingress-nginx-admission-patch--1-nfp8h 0/1 Error 2 (22s ago) 23s default pod/test-ingress-ingress-nginx-controller-98f5696c9-m8k84 1/1 Running 0 23s kube-system pod/coredns-6cff99dc8c-bpv9g 1/1 Running 0 28m kube-system pod/coredns-6cff99dc8c-nv7gj 1/1 Running 0 28m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 84s default service/test-ingress-ingress-nginx-controller LoadBalancer 10.0.0.133 <pending> 80:32264/TCP,443:32554/TCP 23s default service/test-ingress-ingress-nginx-controller-admission ClusterIP 10.0.0.210 <none> 443/TCP 23s kube-system service/kube-dns ClusterIP 10.0.0.53 <none> 53/UDP,53/TCP,9153/TCP 6d2h NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/test-ingress-ingress-nginx-controller 1/1 1 1 23s kube-system deployment.apps/coredns 2/2 2 2 6d2h NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/test-ingress-ingress-nginx-controller-98f5696c9 1 1 1 23s kube-system replicaset.apps/coredns-6cff99dc8c 2 2 2 6d2h NAMESPACE NAME COMPLETIONS DURATION AGE default job.batch/test-ingress-ingress-nginx-admission-patch 0/1 23s 23s sh-5.1$ kubectl logs pod/test-ingress-ingress-nginx-admission-patch--1-nfp8h W0803 03:16:06.529542 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. {"level":"info","msg":"patching webhook configurations 'test-ingress-ingress-nginx-admission' mutating=false, validating=true, failurePolicy=Fail","source":"k8s/k8s.go:39","time":"2021-08-03T03:16:06Z"} {"err":"the server could not find the requested resource","level":"fatal","msg":"failed getting validating webhook","source":"k8s/k8s.go:48","time":"2021-08-03T03:16:06Z"}It works if you do this.
$ helm uninstall test-ingress $ kubectl delete -n default job --all sh-5.1$ cat ingress-nginx.values.yaml controller: image: tag: "v1.0.0-beta.1" digest: "sha256:f058f3fdc940095957695829745956c6acddcaef839907360965e27fd3348e2e" admissionWebhooks: patch: enabled: false sh-5.1$ helm install test-ingress ./charts/ingress-nginx/ --values ingress-nginx.values.yaml NAME: test-ingress LAST DEPLOYED: Tue Aug 3 13:17:48 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the LoadBalancer IP to be available. You can watch the status by running 'kubectl --namespace default get services -o wide -w test-ingress-ingress-nginx-controller' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: name: example namespace: foo spec: rules: - host: www.example.com http: paths: - backend: serviceName: exampleService servicePort: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: <base64 encoded cert> tls.key: <base64 encoded key> type: kubernetes.io/tls sh-5.1$ kubectl get all -A NAMESPACE NAME READY STATUS RESTARTS AGE default pod/test-ingress-ingress-nginx-controller-98f5696c9-z8xnb 0/1 Running 0 5s kube-system pod/coredns-6cff99dc8c-bpv9g 1/1 Running 0 29m kube-system pod/coredns-6cff99dc8c-nv7gj 1/1 Running 0 29m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3m5s default service/test-ingress-ingress-nginx-controller LoadBalancer 10.0.0.103 <pending> 80:31922/TCP,443:31545/TCP 5s default service/test-ingress-ingress-nginx-controller-admission ClusterIP 10.0.0.10 <none> 443/TCP 5s kube-system service/kube-dns ClusterIP 10.0.0.53 <none> 53/UDP,53/TCP,9153/TCP 6d2h NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/test-ingress-ingress-nginx-controller 0/1 1 0 5s kube-system deployment.apps/coredns 2/2 2 2 6d2h NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/test-ingress-ingress-nginx-controller-98f5696c9 1 1 0 5s kube-system replicaset.apps/coredns-6cff99dc8c 2 2 2 6d2hAnything else we need to know:
/kind bug
The text was updated successfully, but these errors were encountered: