Issues with Ingress Configuration and Client Certificate Transmission

[sajith-madhusanka]

What happened:

Despite configuring the nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream annotation to true [1], there is an observed issue where the Ingress fails to transmit the client certificate in an HTTP header. Kindly refer to the trace log extracted from the application. As we can see, the client certificate is not sent to the upstream server.

2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "GET /pizzashack/1.0.0/menu HTTP/1.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Host: gateway.am.wso2.com[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Request-ID: 1c0b142bcec0cc47e029d2c2c5845ebb[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Real-IP: 192.168.49.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-For: 192.168.49.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Host: gateway.am.wso2.com[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Port: 443[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Proto: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Scheme: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Scheme: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "accept: application/json[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Authorization: Bearer xxxxx[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "User-Agent: PostmanRuntime/7.36.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Postman-Token: 5d207db3-8a6a-40a0-924e-b95af4a2a0f2[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Accept-Encoding: gzip, deflate, br[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "[\r][\n]"

What you expected to happen:

According to the documentation [1], the Ingress is expected to send the client certificate in the HTTP header (ssl-client-cert) to the upstream server. This behavior should occur when the Ingress is configured with the 'nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream' annotation as illustrated below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ template "am-single-node.resource.prefix" . }}-am-gateway-ingress
  namespace : {{ .Release.Namespace }}
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - {{ .Values.wso2.deployment.am.ingress.gateway.hostname }}
  rules:
  - host: {{ .Values.wso2.deployment.am.ingress.gateway.hostname }}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ template "am-single-node.resource.prefix" . }}-am-service
            port:
              number: 8243

[1] https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-certificate-authentication

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.9.4
Build: 846d251
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

---

Kubernetes version (use kubectl version):

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Environment:

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS

• Kernel (e.g. uname -a): Linux madhusanka 6.5.0-15-generic Split implementations from generic code #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: minikube

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version

  Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

• kubectl get nodes -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
minikube   Ready    control-plane   51d   v1.28.3   192.168.49.2   <none>        Ubuntu 22.04.3 LTS   6.5.0-15-generic   docker://24.0.7

• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
    minikube addons enable ingress
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

Current State of the controller:

• kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx

Events:

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

NAMESPACE       NAME                                            READY   STATUS      RESTARTS       AGE     IP             NODE       NOMINATED NODE   READINESS GATES
ingress-nginx   pod/ingress-nginx-admission-create-x7sc5        0/1     Completed   0              53d     <none>         minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-admission-patch-2d2px         0/1     Completed   2              53d     <none>         minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-controller-7c6974c4d8-rnw4b   1/1     Running     5 (40h ago)    5d23h   10.244.0.150   minikube   <none>           <none>
kube-system     pod/coredns-5dd5756b68-5vlp2                    1/1     Running     16 (40h ago)   53d     10.244.0.149   minikube   <none>           <none>
kube-system     pod/etcd-minikube                               1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-apiserver-minikube                     1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-controller-manager-minikube            1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-proxy-kshj7                            1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-scheduler-minikube                     1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/storage-provisioner                         1/1     Running     46 (2m ago)    53d     192.168.49.2   minikube   <none>           <none>
test            pod/sample-deployment-c6fbc8fd8-bvmnp           1/1     Running     0              73s     10.244.0.151   minikube   <none>           <none>

NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      53d   <none>
ingress-nginx   service/ingress-nginx-controller             NodePort    10.100.120.112   <none>        80:31217/TCP,443:32365/TCP   53d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.96.252.44     <none>        443/TCP                      53d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       53d   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                               SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   53d   kube-proxy   registry.k8s.io/kube-proxy:v1.28.3   k8s-app=kube-proxy

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS          IMAGES                                                                                                                    SELECTOR
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           53d   controller          registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     deployment.apps/coredns                    1/1     1            1           53d   coredns             registry.k8s.io/coredns/coredns:v1.10.1                                                                                   k8s-app=kube-dns
test            deployment.apps/sample-deployment          1/1     1            1           73s   sample-deployment   mendhak/http-https-echo:31                                                                                                app=sample-deployment

NAMESPACE       NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS          IMAGES                                                                                                                    SELECTOR
ingress-nginx   replicaset.apps/ingress-nginx-controller-7c6974c4d8   1         1         1       53d   controller          registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7c6974c4d8
kube-system     replicaset.apps/coredns-5dd5756b68                    1         1         1       53d   coredns             registry.k8s.io/coredns/coredns:v1.10.1                                                                                   k8s-app=kube-dns,pod-template-hash=5dd5756b68
test            replicaset.apps/sample-deployment-c6fbc8fd8           1         1         1       73s   sample-deployment   mendhak/http-https-echo:31                                                                                                app=sample-deployment,pod-template-hash=c6fbc8fd8

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES                                                                                                                                           SELECTOR
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           27s        53d   create       registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80   batch.kubernetes.io/controller-uid=7a2a9743-64f7-43a6-a94a-aee8b824772a
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           40s        53d   patch        registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80   batch.kubernetes.io/controller-uid=5cd1241e-b616-450e-b058-88e77c303f11

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             ingress-nginx-controller-7c6974c4d8-rnw4b
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Mon, 29 Jan 2024 16:12:44 +0530
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=7c6974c4d8
Annotations:      <none>
Status:           Running
IP:               10.244.0.150
IPs:
  IP:           10.244.0.150
Controlled By:  ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  docker://1a1719785529d2a617cde7e9c10c4869168cedbd7a6160aebc74f11c02b6f78e
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 04 Feb 2024 15:32:36 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 02 Feb 2024 20:15:41 +0530
      Finished:     Fri, 02 Feb 2024 23:16:21 +0530
    Ready:          True
    Restart Count:  5
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7c6974c4d8-rnw4b (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x6hr9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-x6hr9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age                    From                      Message
  ----    ------          ----                   ----                      -------
  Normal  RELOAD          40h (x8 over 43h)      nginx-ingress-controller  NGINX reload triggered due to a change in configuration
  Normal  SandboxChanged  6m58s                  kubelet                   Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m58s                  kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal  Created         6m58s                  kubelet                   Created container controller
  Normal  Started         6m58s                  kubelet                   Started container controller
  Normal  RELOAD          5m32s (x2 over 6m56s)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
Annotations:              <none>
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.120.112
IPs:                      10.100.120.112
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31217/TCP
Endpoints:                10.244.0.150:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32365/TCP
Endpoints:                10.244.0.150:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Current state of ingress object, if applicable:

• kubectl -n <appnamespace> get all,ing -o wide

NAME                                    READY   STATUS    RESTARTS   AGE    IP             NODE       NOMINATED NODE   READINESS GATES
pod/sample-deployment-c6fbc8fd8-bvmnp   1/1     Running   0          8m8s   10.244.0.151   minikube   <none>           <none>

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS          IMAGES                       SELECTOR
deployment.apps/sample-deployment   1/1     1            1           8m8s   sample-deployment   mendhak/http-https-echo:31   app=sample-deployment

NAME                                          DESIRED   CURRENT   READY   AGE    CONTAINERS          IMAGES                       SELECTOR
replicaset.apps/sample-deployment-c6fbc8fd8   1         1         1       8m8s   sample-deployment   mendhak/http-https-echo:31   app=sample-deployment,pod-template-hash=c6fbc8fd8

NAME                                       CLASS   HOSTS                 ADDRESS        PORTS     AGE
ingress.networking.k8s.io/sample-ingress   nginx   gateway.am.wso2.com   192.168.49.2   80, 443   7m59s

• kubectl -n <appnamespace> describe ing <ingressname>

Name:             sample-ingress
Labels:           <none>
Namespace:        test
Address:          192.168.49.2
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes gateway.am.wso2.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  gateway.am.wso2.com  
                       /   sample-service:8443 (<error: endpoints "sample-service" not found>)
Annotations:           nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: true
                       nginx.ingress.kubernetes.io/auth-tls-verify-client: optional_no_ca
                       nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    8m45s (x2 over 9m21s)  nginx-ingress-controller  Scheduled for sync

• If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

• Install minikube
• Install nginx ingress controller
  minikube addons enable ingress
• Deploy an application that echoes the request along with headers.

kubectl create namespace test
kubectl apply -f sample-deployment.yaml

sample-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deployment
  namespace: test
spec:
  replicas: 1  # Adjust the number of replicas as needed
  selector:
    matchLabels:
      app: sample-deployment
  template:
    metadata:
      labels:
        app: sample-deployment
    spec:
      containers:
      - name: sample-deployment
        image: mendhak/http-https-echo:31  # Use the image name from your Docker build
        ports:
        - containerPort: 8080
        - containerPort: 8443

• Deploy a k8s service
  kubectl apply -f sample-service.yaml

sample-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: sample-service
  namespace : test
spec:
  # label keys and values that must match in order to receive traffic for this service
  selector:
    app: sample-deployment
  ports:
    # ports that this service should serve on
    - name: pass-through-http
      protocol: TCP
      port: 8080
    - name: pass-through-https
      protocol: TCP
      port: 8443

• Deploy an ingress
  kubectl apply -f sample-Ing.yaml

sample-Ing.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-ingress
  namespace : test
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - gateway.am.wso2.com
  rules:
  - host: gateway.am.wso2.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sample-service
            port:
              number: 8443

• Execute the below curl command to reproduce the issue.

curl --cert public.crt --key client.key -X 'GET'   'https://gateway.am.wso2.com/pizzashack/1.0.0/menu'   -H 'accept: application/json'  -k

• Please find the client.key and public.crt to use in the above curl command. Please remove the .txt extension after downloading the files below.

client .key.txt
public.crt.txt

• Tail the logs of the echo application (sample-deployment) and check the headers. As we can see, it doesn't send the client certificate in a HTTP header to upstream server.

{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "x-request-id": "481546ef2cb576066fdfa22f7e373edb",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}
}

Anything else we need to know:

• what is the command you used to start minikube
  minikube start

😄  minikube v1.32.0 on Ubuntu 22.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.9.4
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

• what is the --driver for minikube
  docker
• what is the command and output of nslookup gateway.am.wso2.com

nslookup gateway.am.wso2.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Name:	gateway.am.wso2.com
Address: 192.168.49.2

Please note that I've added the following host mapping to the /etc/hosts file on my local machine (host machine).
192.168.49.2 gateway.am.wso2.com

• what is the command and output of minikube ip

192.168.49.2

• what is the command and output of kubectl cluster-info

Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

• what is the command and output of kubectl get svc -A | grep -i ingress

ingress-nginx   ingress-nginx-controller             NodePort    10.100.120.112   <none>        80:31217/TCP,443:32365/TCP   53d
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.96.252.44     <none>        443/TCP                      53d

Thanks & Regards
Sajith M

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

@sajith-madhusanka if this is true then a working feature broke but I don't recall an PR related to the feature

• Can you please remove the images and replace it with the commands and outputs copy/pasted
• Can you try a curl command without postman but curl cli and send cert etc so that a reader can try to get hints for reproducing
• The screenshot does not show the external-ip of minikube etc so once clear on the use-case, maybe we can try to reproduce together on a screen ssharing session with you

/remove-kind bug
/triage needs-information

[sajith-madhusanka]

Hi @longwuyuan

I've updated the issue description as suggested by removing the screenshots.

Please find the curl command to reproduce the issue.

curl --cert public.crt --key client.key -X 'GET' 'https://gateway.am.wso2.com/pizzashack/1.0.0/menu' -H 'accept: application/json' -k
Please find the client.key and public.crt to use in the above curl command. Please remove the .txt extension after downloading the files below.

client .key.txt
public.crt.txt

Please let me know if you need further information.

Thanks & Regards
Sajith M

[longwuyuan]

• Since the ingress-nginx controller service is of type NodePort, the curl command destination you show is invalid, AFAIK.
• The reason is that the destination should be <ipaddress:<NodePort> like <minikube-ip>:32365 with the protocol as HTTPS
• I think you can lookup metallb.org and implement that. Use your OS VM software as minikube driver and use the ipaddress of the VM to configure the metallb pool

[sajith-madhusanka]

Hi @longwuyuan

With the shared curl command, I'm accessing a deployment (which is not publicly accessible) that is deployed on my local machine, and I can access it without any issues. Please refer to the curl command output below,

curl --cert public.crt --key client.key -X 'GET' \
  'https://gateway.am.wso2.com/pizzashack/1.0.0/menu' \
  -H 'accept: application/json'  -kv
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.49.2:443...
* Connected to gateway.am.wso2.com (192.168.49.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb  4 10:02:37 2024 GMT
*  expire date: Feb  3 10:02:37 2025 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55d908886e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /pizzashack/1.0.0/menu HTTP/2
> Host: gateway.am.wso2.com
> user-agent: curl/7.81.0
> accept: application/json
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< date: Sun, 04 Feb 2024 12:16:50 GMT
< content-type: application/json; charset=utf-8
< content-length: 848
< x-powered-by: Express
< etag: W/"350-k4xI6rVSI+/HtvU9TgZpAFYV2fw"
< 
{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "x-request-id": "52bde7a617e6535a8bb874fa5b35bd94",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host gateway.am.wso2.com left intact
}

Hope you can follow the steps stated under the "How to reproduce this issue:" section in the issue description to deploy a similar setup on your local machine and reproduce the issue. If it is required to have a screen-sharing session to further discuss the issue, we can schedule it for this Tuesday (6th of February 2024).

Thanks & Regards
Sajith M

[longwuyuan]

• Since the ingress-nginx controller service is of type NodePort, the curl command destination you show is invalid, AFAIK.
• The reason the curl command is invalid is that the destination should be <ipaddress: like :32365, with the protocol as HTTPS. This is a complicated networking
• what is the command you used to start minikube
• what is the --driver for minikube
• what is the command and output of nslookup gateway.am.wso2.com
• what is the command and output of minikube ip
• what is the command and output of kubectl cluster-info
• what is the command and output of kubectl get svc -A | grep -i ingress
• Request you to edit the issue description with the information on the problem and not repeat the information once provided because not sure which one to use for reproducing

[sajith-madhusanka]

Hi @longwuyuan

I've updated the issue description with the required information. You can find that under the 'Anything else we need to know:' section.

Thanks & Regards
Sajith M

[longwuyuan]

• Either send your request to the NodePort port 3235 with protocol HTTPS and show complate details like
  • curl command and output with -v
  • logs of controller pod
  • logs of app pod
• Or deploy another application like httpbun.com over TLS in namespace default and show same above details
• Or switch to a VM driver for minikube and then show all those details
• show proof that this host gateway.am.wso2.com is listening on port 443