Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single-service ingress not recognizing ssl-passthrough #11829

Open
kjyo6 opened this issue Aug 19, 2024 · 10 comments
Open

Single-service ingress not recognizing ssl-passthrough #11829

kjyo6 opened this issue Aug 19, 2024 · 10 comments
Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@kjyo6
Copy link

kjyo6 commented Aug 19, 2024

What happened:

We tried using this technique https://kubernetes.io/docs/concepts/services-networking/ingress/#single-service-ingress to send all ingress requests to a single back-end service. The backend service is authenticated using client certificates. We enabled SSL passthrough on the ingress controller via the startup flags. When browsing to the ingress route, we are served the correct certificate from the backend server but we see the client certificate being dropped. This seems like a bug and is not documented anywhere that it should or should not work.

What you expected to happen:

The Ingress Controller's ssl-passthrough is not working

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.11.1
Build: 7c44f99
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6", GitCommit:"11902a838028edef305dfe2f96be929bc4d114d8", GitTreeState:"clean", BuildDate:"2023-06-14T09:56:58Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"30+", GitVersion:"v1.30.2-eks-db838b0", GitCommit:"04088714581f0ad0a9e2c81c6ecc36bdd30d4b53", GitTreeState:"clean", BuildDate:"2024-06-27T19:09:53Z", GoVersion:"go1.22.4", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.26) and server (1.30) exceeds the supported minor version skew of +/-1

Environment:

AWS EKS, RHEL, AMZN Linux 2
Installed EKS and its node groups using our custom AMI and used our cloudformation templates to create/update node groups

EKS version 1.30, we have 4 worker nodes with latest containerd, EKS and Amazon Linux 2 versions

Name:         ingclass-app
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-app
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.11.1
Annotations:  <none>
Controller:   k8s.io/ingress-app
Events:       <none>

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide

  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

  • Current state of ingress object, if applicable:

    • kubectl -n <appnamespace> get all,ing -o wide
      Don't have permissions to show but I want to say that we have running apps and I checked their logs
    • kubectl -n <appnamespace> describe ing <ingressname>
      I have an ingress app that's running with rules (hosts, path, ports, and annotations such as nginx.ingress.kubernetes.io/backend-protocol: https and nginx.ingress.kubernetes.io/ssl-passthrough: true
    • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help

Anything else we need to know:

Once I refresh the app more than 10 times, I am getting the error that says: 2024/01/01 00:00:00 [info] 21#21: *1609 peer closed connection in SSL handshake while SSL handshaking, client: ..., server: 0.0.0.0:*port number, then I get several 403 log messages. Everything gets back to normal after a minute

@kjyo6 kjyo6 added the kind/bug Categorizes issue or PR as related to a bug. label Aug 19, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Aug 19, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@longwuyuan
Copy link
Contributor

/remove-kind bug

Edit the issue description and answer all the questions that are asked in the new bug report template because readers need the small tiny details to do any analysis. The details are also critical to being able to reproduce in a minikube or a kind cluster.

Since you are terminating SSL on the backend pod and not in the controller, I am not sure why you would think that the intricate handshake and connection process for SSL has much to do with the controller.

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Aug 20, 2024
@longwuyuan
Copy link
Contributor

/kind support

@k8s-ci-robot k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Aug 20, 2024
@kjyo6
Copy link
Author

kjyo6 commented Aug 20, 2024

@longwuyuan I updated my info based on my permission I have. Let me know if you need more.

Also I am investigating nginx conf file and see the location defined in the file.

@longwuyuan
Copy link
Contributor

I see you updated some text. But that text can not be analyzed.

You don't need to provide info from your prod cluster. You can create a cluster a kind cluster or a minikube cluster and then show all the small tiny intricate detailed information of a test from that kind/minikube cluster.

@kjyo6
Copy link
Author

kjyo6 commented Aug 20, 2024

Unfortunately I cannot reproduce this on kind or Minikube, but instead I have the /api location that fail in the middle of loading the web page, I am getting 403 errors for several /api/... GET's. I am wondering if there are several failing 403 GETs then is it more likely to be a controller problem or pod problem? I am looking back the previous version, probably before version 1.9.6, since I didn't see this error a few months ago.

/remove-kind bug
/kind support

@k8s-ci-robot
Copy link
Contributor

@kjyo6: Those labels are not set on the issue: kind/bug

In response to this:

Unfortunately I cannot reproduce this on kind or Minikube, but instead I have the /api location that fail in the middle of loading the web page, I am getting 403 errors for several /api/... GET's. I am wondering if there are several failing 403 GETs then is it more likely to be a controller problem or pod problem? I am looking back the previous version, probably before version 1.9.6, since I didn't see this error a few months ago.

/remove-kind bug
/kind support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kjyo6
Copy link
Author

kjyo6 commented Aug 20, 2024

/remove-kind support

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/support Categorizes issue or PR as a support question. labels Aug 20, 2024
@kjyo6
Copy link
Author

kjyo6 commented Aug 20, 2024

Also I am hoping that my issue will be solved with this fix. I am following up with this fix: #11498

Copy link

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

@github-actions github-actions bot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
Development

No branches or pull requests

3 participants