Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues with Ingress Configuration and Client Certificate Transmission #10967

Closed
sajith-madhusanka opened this issue Feb 2, 2024 · 15 comments
Closed

Issues with Ingress Configuration and Client Certificate Transmission #10967

sajith-madhusanka opened this issue Feb 2, 2024 · 15 comments
Labels
needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@sajith-madhusanka
Copy link

sajith-madhusanka commented Feb 2, 2024 •
edited
Loading

What happened:

Despite configuring the nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream annotation to true [1], there is an observed issue where the Ingress fails to transmit the client certificate in an HTTP header. Kindly refer to the trace log extracted from the application. As we can see, the client certificate is not sent to the upstream server.

2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "GET /pizzashack/1.0.0/menu HTTP/1.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Host: gateway.am.wso2.com[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Request-ID: 1c0b142bcec0cc47e029d2c2c5845ebb[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Real-IP: 192.168.49.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-For: 192.168.49.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Host: gateway.am.wso2.com[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Port: 443[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Proto: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Forwarded-Scheme: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "X-Scheme: https[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "accept: application/json[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Authorization: Bearer xxxxx[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "User-Agent: PostmanRuntime/7.36.1[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Postman-Token: 5d207db3-8a6a-40a0-924e-b95af4a2a0f2[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "Accept-Encoding: gzip, deflate, br[\r][\n]"
[2024-02-02 15:17:22,717] DEBUG - wire HTTPS-Listener I/O dispatcher-9 >> "[\r][\n]"

What you expected to happen:

According to the documentation [1], the Ingress is expected to send the client certificate in the HTTP header (ssl-client-cert) to the upstream server. This behavior should occur when the Ingress is configured with the 'nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream' annotation as illustrated below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ template "am-single-node.resource.prefix" . }}-am-gateway-ingress
  namespace : {{ .Release.Namespace }}
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - {{ .Values.wso2.deployment.am.ingress.gateway.hostname }}
  rules:
  - host: {{ .Values.wso2.deployment.am.ingress.gateway.hostname }}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ template "am-single-node.resource.prefix" . }}-am-service
            port:
              number: 8243

[1] https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#client-certificate-authentication

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.9.4
Build: 846d251
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Environment:

  • Cloud provider or hardware configuration:

  • OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS

  • Kernel (e.g. uname -a): Linux madhusanka 6.5.0-15-generic Split implementations from generic code #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools: minikube

    • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

  • Basic cluster related info:

    • kubectl version
  Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
  • kubectl get nodes -o wide
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
minikube   Ready    control-plane   51d   v1.28.3   192.168.49.2   <none>        Ubuntu 22.04.3 LTS   6.5.0-15-generic   docker://24.0.7
  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress
    • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
    • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
      minikube addons enable ingress
    • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

Current State of the controller:

  • kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx

Events:

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
NAMESPACE       NAME                                            READY   STATUS      RESTARTS       AGE     IP             NODE       NOMINATED NODE   READINESS GATES
ingress-nginx   pod/ingress-nginx-admission-create-x7sc5        0/1     Completed   0              53d     <none>         minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-admission-patch-2d2px         0/1     Completed   2              53d     <none>         minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-controller-7c6974c4d8-rnw4b   1/1     Running     5 (40h ago)    5d23h   10.244.0.150   minikube   <none>           <none>
kube-system     pod/coredns-5dd5756b68-5vlp2                    1/1     Running     16 (40h ago)   53d     10.244.0.149   minikube   <none>           <none>
kube-system     pod/etcd-minikube                               1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-apiserver-minikube                     1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-controller-manager-minikube            1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-proxy-kshj7                            1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-scheduler-minikube                     1/1     Running     15 (40h ago)   53d     192.168.49.2   minikube   <none>           <none>
kube-system     pod/storage-provisioner                         1/1     Running     46 (2m ago)    53d     192.168.49.2   minikube   <none>           <none>
test            pod/sample-deployment-c6fbc8fd8-bvmnp           1/1     Running     0              73s     10.244.0.151   minikube   <none>           <none>

NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      53d   <none>
ingress-nginx   service/ingress-nginx-controller             NodePort    10.100.120.112   <none>        80:31217/TCP,443:32365/TCP   53d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.96.252.44     <none>        443/TCP                      53d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       53d   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                               SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   53d   kube-proxy   registry.k8s.io/kube-proxy:v1.28.3   k8s-app=kube-proxy

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS          IMAGES                                                                                                                    SELECTOR
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           53d   controller          registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     deployment.apps/coredns                    1/1     1            1           53d   coredns             registry.k8s.io/coredns/coredns:v1.10.1                                                                                   k8s-app=kube-dns
test            deployment.apps/sample-deployment          1/1     1            1           73s   sample-deployment   mendhak/http-https-echo:31                                                                                                app=sample-deployment

NAMESPACE       NAME                                                  DESIRED   CURRENT   READY   AGE   CONTAINERS          IMAGES                                                                                                                    SELECTOR
ingress-nginx   replicaset.apps/ingress-nginx-controller-7c6974c4d8   1         1         1       53d   controller          registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7c6974c4d8
kube-system     replicaset.apps/coredns-5dd5756b68                    1         1         1       53d   coredns             registry.k8s.io/coredns/coredns:v1.10.1                                                                                   k8s-app=kube-dns,pod-template-hash=5dd5756b68
test            replicaset.apps/sample-deployment-c6fbc8fd8           1         1         1       73s   sample-deployment   mendhak/http-https-echo:31                                                                                                app=sample-deployment,pod-template-hash=c6fbc8fd8

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES                                                                                                                                           SELECTOR
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           27s        53d   create       registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80   batch.kubernetes.io/controller-uid=7a2a9743-64f7-43a6-a94a-aee8b824772a
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           40s        53d   patch        registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80   batch.kubernetes.io/controller-uid=5cd1241e-b616-450e-b058-88e77c303f11
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
Name:             ingress-nginx-controller-7c6974c4d8-rnw4b
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Mon, 29 Jan 2024 16:12:44 +0530
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=7c6974c4d8
Annotations:      <none>
Status:           Running
IP:               10.244.0.150
IPs:
  IP:           10.244.0.150
Controlled By:  ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  docker://1a1719785529d2a617cde7e9c10c4869168cedbd7a6160aebc74f11c02b6f78e
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 04 Feb 2024 15:32:36 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 02 Feb 2024 20:15:41 +0530
      Finished:     Fri, 02 Feb 2024 23:16:21 +0530
    Ready:          True
    Restart Count:  5
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7c6974c4d8-rnw4b (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x6hr9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-x6hr9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age                    From                      Message
  ----    ------          ----                   ----                      -------
  Normal  RELOAD          40h (x8 over 43h)      nginx-ingress-controller  NGINX reload triggered due to a change in configuration
  Normal  SandboxChanged  6m58s                  kubelet                   Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          6m58s                  kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" already present on machine
  Normal  Created         6m58s                  kubelet                   Created container controller
  Normal  Started         6m58s                  kubelet                   Started container controller
  Normal  RELOAD          5m32s (x2 over 6m56s)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
Annotations:              <none>
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.120.112
IPs:                      10.100.120.112
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31217/TCP
Endpoints:                10.244.0.150:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32365/TCP
Endpoints:                10.244.0.150:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP             NODE       NOMINATED NODE   READINESS GATES
pod/sample-deployment-c6fbc8fd8-bvmnp   1/1     Running   0          8m8s   10.244.0.151   minikube   <none>           <none>

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS          IMAGES                       SELECTOR
deployment.apps/sample-deployment   1/1     1            1           8m8s   sample-deployment   mendhak/http-https-echo:31   app=sample-deployment

NAME                                          DESIRED   CURRENT   READY   AGE    CONTAINERS          IMAGES                       SELECTOR
replicaset.apps/sample-deployment-c6fbc8fd8   1         1         1       8m8s   sample-deployment   mendhak/http-https-echo:31   app=sample-deployment,pod-template-hash=c6fbc8fd8

NAME                                       CLASS   HOSTS                 ADDRESS        PORTS     AGE
ingress.networking.k8s.io/sample-ingress   nginx   gateway.am.wso2.com   192.168.49.2   80, 443   7m59s
  • kubectl -n <appnamespace> describe ing <ingressname>
Name:             sample-ingress
Labels:           <none>
Namespace:        test
Address:          192.168.49.2
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes gateway.am.wso2.com
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  gateway.am.wso2.com  
                       /   sample-service:8443 (<error: endpoints "sample-service" not found>)
Annotations:           nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: true
                       nginx.ingress.kubernetes.io/auth-tls-verify-client: optional_no_ca
                       nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    8m45s (x2 over 9m21s)  nginx-ingress-controller  Scheduled for sync

  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help

How to reproduce this issue:

  • Install minikube
  • Install nginx ingress controller
    minikube addons enable ingress
  • Deploy an application that echoes the request along with headers.

kubectl create namespace test
kubectl apply -f sample-deployment.yaml

sample-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deployment
  namespace: test
spec:
  replicas: 1  # Adjust the number of replicas as needed
  selector:
    matchLabels:
      app: sample-deployment
  template:
    metadata:
      labels:
        app: sample-deployment
    spec:
      containers:
      - name: sample-deployment
        image: mendhak/http-https-echo:31  # Use the image name from your Docker build
        ports:
        - containerPort: 8080
        - containerPort: 8443
  • Deploy a k8s service
    kubectl apply -f sample-service.yaml

sample-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: sample-service
  namespace : test
spec:
  # label keys and values that must match in order to receive traffic for this service
  selector:
    app: sample-deployment
  ports:
    # ports that this service should serve on
    - name: pass-through-http
      protocol: TCP
      port: 8080
    - name: pass-through-https
      protocol: TCP
      port: 8443
  • Deploy an ingress
    kubectl apply -f sample-Ing.yaml

sample-Ing.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-ingress
  namespace : test
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - gateway.am.wso2.com
  rules:
  - host: gateway.am.wso2.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sample-service
            port:
              number: 8443
  • Execute the below curl command to reproduce the issue.
curl --cert public.crt --key client.key -X 'GET'   'https://gateway.am.wso2.com/pizzashack/1.0.0/menu'   -H 'accept: application/json'  -k
  • Please find the client.key and public.crt to use in the above curl command. Please remove the .txt extension after downloading the files below.

client .key.txt
public.crt.txt

  • Tail the logs of the echo application (sample-deployment) and check the headers. As we can see, it doesn't send the client certificate in a HTTP header to upstream server.
{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "x-request-id": "481546ef2cb576066fdfa22f7e373edb",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}
}

Anything else we need to know:

  • what is the command you used to start minikube
    minikube start
😄  minikube v1.32.0 on Ubuntu 22.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.9.4
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
🔎  Verifying ingress addon...
🌟  Enabled addons: storage-provisioner, default-storageclass, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
  • what is the --driver for minikube
    docker
  • what is the command and output of nslookup gateway.am.wso2.com
nslookup gateway.am.wso2.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Name:	gateway.am.wso2.com
Address: 192.168.49.2

Please note that I've added the following host mapping to the /etc/hosts file on my local machine (host machine).
192.168.49.2 gateway.am.wso2.com

  • what is the command and output of minikube ip
192.168.49.2
  • what is the command and output of kubectl cluster-info
Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
  • what is the command and output of kubectl get svc -A | grep -i ingress

ingress-nginx   ingress-nginx-controller             NodePort    10.100.120.112   <none>        80:31217/TCP,443:32365/TCP   53d
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.96.252.44     <none>        443/TCP                      53d

Thanks & Regards
Sajith M
@sajith-madhusanka sajith-madhusanka added the kind/bug Categorizes issue or PR as related to a bug. label Feb 2, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Feb 2, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@longwuyuan
Copy link
Contributor

@sajith-madhusanka if this is true then a working feature broke but I don't recall an PR related to the feature

  • Can you please remove the images and replace it with the commands and outputs copy/pasted
  • Can you try a curl command without postman but curl cli and send cert etc so that a reader can try to get hints for reproducing
  • The screenshot does not show the external-ip of minikube etc so once clear on the use-case, maybe we can try to reproduce together on a screen ssharing session with you

/remove-kind bug
/triage needs-information

@k8s-ci-robot k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Feb 3, 2024
@sajith-madhusanka
Copy link
Author

sajith-madhusanka commented Feb 4, 2024 •
edited
Loading

Hi @longwuyuan

I've updated the issue description as suggested by removing the screenshots.

Please find the curl command to reproduce the issue.

curl --cert public.crt --key client.key -X 'GET' 'https://gateway.am.wso2.com/pizzashack/1.0.0/menu' -H 'accept: application/json' -k
Please find the client.key and public.crt to use in the above curl command. Please remove the .txt extension after downloading the files below.

client .key.txt
public.crt.txt

Please let me know if you need further information.

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

  • Since the ingress-nginx controller service is of type NodePort, the curl command destination you show is invalid, AFAIK.
  • The reason is that the destination should be <ipaddress:<NodePort> like <minikube-ip>:32365 with the protocol as HTTPS
  • I think you can lookup metallb.org and implement that. Use your OS VM software as minikube driver and use the ipaddress of the VM to configure the metallb pool

@sajith-madhusanka
Copy link
Author

sajith-madhusanka commented Feb 4, 2024 •
edited
Loading

Hi @longwuyuan

With the shared curl command, I'm accessing a deployment (which is not publicly accessible) that is deployed on my local machine, and I can access it without any issues. Please refer to the curl command output below,

curl --cert public.crt --key client.key -X 'GET' \
  'https://gateway.am.wso2.com/pizzashack/1.0.0/menu' \
  -H 'accept: application/json'  -kv
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.49.2:443...
* Connected to gateway.am.wso2.com (192.168.49.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb  4 10:02:37 2024 GMT
*  expire date: Feb  3 10:02:37 2025 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55d908886e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /pizzashack/1.0.0/menu HTTP/2
> Host: gateway.am.wso2.com
> user-agent: curl/7.81.0
> accept: application/json
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< date: Sun, 04 Feb 2024 12:16:50 GMT
< content-type: application/json; charset=utf-8
< content-length: 848
< x-powered-by: Express
< etag: W/"350-k4xI6rVSI+/HtvU9TgZpAFYV2fw"
< 
{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "x-request-id": "52bde7a617e6535a8bb874fa5b35bd94",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host gateway.am.wso2.com left intact
}

Hope you can follow the steps stated under the "How to reproduce this issue:" section in the issue description to deploy a similar setup on your local machine and reproduce the issue. If it is required to have a screen-sharing session to further discuss the issue, we can schedule it for this Tuesday (6th of February 2024).

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

  • Since the ingress-nginx controller service is of type NodePort, the curl command destination you show is invalid, AFAIK.
  • The reason the curl command is invalid is that the destination should be <ipaddress: like :32365, with the protocol as HTTPS. This is a complicated networking
  • what is the command you used to start minikube
  • what is the --driver for minikube
  • what is the command and output of nslookup gateway.am.wso2.com
  • what is the command and output of minikube ip
  • what is the command and output of kubectl cluster-info
  • what is the command and output of kubectl get svc -A | grep -i ingress
  • Request you to edit the issue description with the information on the problem and not repeat the information once provided because not sure which one to use for reproducing

@sajith-madhusanka
Copy link
Author

Hi @longwuyuan

I've updated the issue description with the required information. You can find that under the 'Anything else we need to know:' section.

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

  • Either send your request to the NodePort port 3235 with protocol HTTPS and show complate details like
    • curl command and output with -v
    • logs of controller pod
    • logs of app pod
  • Or deploy another application like httpbun.com over TLS in namespace default and show same above details
  • Or switch to a VM driver for minikube and then show all those details
  • show proof that this host gateway.am.wso2.com is listening on port 443

@sajith-madhusanka
Copy link
Author

Hi @longwuyuan,

Please find the requested information.

  • curl command and output with -v
curl --cert public.crt --key client.key -X 'GET' 'https://gateway.am.wso2.com/pizzashack/1.0.0/menu'  -H 'accept: application/json' -kv
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.49.2:443...
* Connected to gateway.am.wso2.com (192.168.49.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb  5 03:33:27 2024 GMT
*  expire date: Feb  4 03:33:27 2025 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55acb9c85e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /pizzashack/1.0.0/menu HTTP/2
> Host: gateway.am.wso2.com
> user-agent: curl/7.81.0
> accept: application/json
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< date: Mon, 05 Feb 2024 07:54:10 GMT
< content-type: application/json; charset=utf-8
< content-length: 848
< x-powered-by: Express
< etag: W/"350-OziO/PY5lrUeAPnXPD5MTczBq4k"
< 
{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "x-request-id": "381b4ca3206abc3bab19a6756cfa0cbc",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host gateway.am.wso2.com left intact
  • logs of controller pod
kubectl logs -f ingress-nginx-controller-7c6974c4d8-rnw4b -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0205 03:33:27.755812       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0205 03:33:27.756011       7 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0205 03:33:27.763931       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28" git="v1.28.3" state="clean" commit="a8a1abc25cad87333840cd7d54be2efaf31a3177" platform="linux/amd64"
I0205 03:33:27.851320       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0205 03:33:27.864016       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0205 03:33:27.870970       7 nginx.go:260] "Starting NGINX Ingress controller"
I0205 03:33:27.874113       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"af619812-6b63-4882-869b-ad964125a081", APIVersion:"v1", ResourceVersion:"770234", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0205 03:33:27.876974       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"b196280f-84c3-49c6-8d69-0a0d89297b99", APIVersion:"v1", ResourceVersion:"628", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0205 03:33:27.877003       7 event.go:298] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"d7be1cd3-d11f-4f9c-a5ba-da18ce20eee6", APIVersion:"v1", ResourceVersion:"629", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0205 03:33:28.973685       7 store.go:440] "Found valid IngressClass" ingress="test/sample-ingress" ingressclass="nginx"
I0205 03:33:28.973819       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"sample-ingress", UID:"47acf45f-a623-4a1d-99bb-507ef06803e1", APIVersion:"networking.k8s.io/v1", ResourceVersion:"841912", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0205 03:33:29.072791       7 nginx.go:303] "Starting NGINX process"
I0205 03:33:29.072929       7 leaderelection.go:245] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0205 03:33:29.073768       7 nginx.go:323] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0205 03:33:29.074396       7 controller.go:190] "Configuration changes detected, backend reload required"
I0205 03:33:29.086239       7 leaderelection.go:255] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0205 03:33:29.086303       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-7c6974c4d8-rnw4b"
I0205 03:33:29.088530       7 status.go:219] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-7c6974c4d8-rnw4b" node="minikube"
I0205 03:33:29.090764       7 status.go:304] "updating Ingress status" namespace="test" ingress="sample-ingress" currentValue=[{"ip":"192.168.49.2"}] newValue=[]
I0205 03:33:29.095521       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"sample-ingress", UID:"47acf45f-a623-4a1d-99bb-507ef06803e1", APIVersion:"networking.k8s.io/v1", ResourceVersion:"843457", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0205 03:33:29.146849       7 controller.go:210] "Backend successfully reloaded"
I0205 03:33:29.146907       7 controller.go:221] "Initial sync, sleeping for 1 second"
I0205 03:33:29.146941       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7c6974c4d8-rnw4b", UID:"f289f191-67a6-4337-9a9a-ab32bb1128f5", APIVersion:"v1", ResourceVersion:"843378", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0205 03:34:29.093777       7 status.go:304] "updating Ingress status" namespace="test" ingress="sample-ingress" currentValue=null newValue=[{"ip":"192.168.49.2"}]
I0205 03:34:29.100710       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"sample-ingress", UID:"47acf45f-a623-4a1d-99bb-507ef06803e1", APIVersion:"networking.k8s.io/v1", ResourceVersion:"843529", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
192.168.49.1 - - [05/Feb/2024:07:54:10 +0000] "GET /pizzashack/1.0.0/menu HTTP/2.0" 200 848 "-" "curl/7.81.0" 59 0.024 [test-sample-service-8443] [] 10.244.0.157:8443 848 0.024 200 381b4ca3206abc3bab19a6756cfa0cbc
  • logs of app pod
kubectl logs -f sample-deployment-c6fbc8fd8-bvmnp -n test
Listening on ports 8080 for http, and 8443 for https.
{
    "path": "/pizzashack/1.0.0/menu",
    "headers": {
        "host": "gateway.am.wso2.com",
        "x-request-id": "381b4ca3206abc3bab19a6756cfa0cbc",
        "x-real-ip": "192.168.49.1",
        "x-forwarded-for": "192.168.49.1",
        "x-forwarded-host": "gateway.am.wso2.com",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https",
        "x-forwarded-scheme": "https",
        "x-scheme": "https",
        "user-agent": "curl/7.81.0",
        "accept": "application/json"
    },
    "method": "GET",
    "body": "",
    "fresh": false,
    "hostname": "gateway.am.wso2.com",
    "ip": "192.168.49.1",
    "ips": [
        "192.168.49.1"
    ],
    "protocol": "https",
    "query": {},
    "subdomains": [
        "am",
        "gateway"
    ],
    "xhr": false,
    "os": {
        "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
    },
    "connection": {
        "servername": false
    },
    "clientCertificate": {}
}
192.168.49.1 - - [05/Feb/2024:07:54:10 +0000] "GET /pizzashack/1.0.0/menu HTTP/1.1" 200 848 "-" "curl/7.81.0"
  • show proof that this host gateway.am.wso2.com is listening on port 443
kubectl get ing -n test
NAME             CLASS   HOSTS                 ADDRESS        PORTS     AGE
sample-ingress   nginx   gateway.am.wso2.com   192.168.49.2   80, 443   17h

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

  • why are you sending your request to localhost port 443 when the ingress-controller service is listening on NodePort 32365
  • Why is there no TLS section in your ingress when your request is a HTTPS request

@sajith-madhusanka
Copy link
Author

Hi @longwuyuan

  • why are you sending your request to localhost port 443 when the ingress-controller service is listening on NodePort 32365

Actually, I'm not sending the request to localhost port 443; instead, that request is going through the ingress controller.
In this case, I've deployed the following Ingress object to enable access to my sample application via a Kubernetes service named 'sample-service.'

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-ingress
  namespace : test
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - gateway.am.wso2.com
  rules:
  - host: gateway.am.wso2.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sample-service
            port:
              number: 8443

After deploying the above Ingress object, it will add the required configuration to the nginx.conf file in the Ingress controller pod. This configuration allows me to access the application deployed in the Kubernetes cluster via 'gateway.am.wso2.com' on port 443.

## start server gateway.am.wso2.com
	server {
		server_name gateway.am.wso2.com ;
		
		listen 80  ;
		listen 443  ssl http2 ;
		
		set $proxy_upstream_name "-";
		
		ssl_certificate_by_lua_block {
			certificate.call()
		}
		
		location / {
			
			set $namespace      "test";
			set $ingress_name   "sample-ingress";
			set $service_name   "sample-service";
			set $service_port   "8443";
			set $location_path  "/";
			set $global_rate_limit_exceeding n;
			
			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					preserve_trailing_slash = false,
					use_port_in_redirects = false,
					global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
				})
				balancer.rewrite()
				plugins.run()
			}
			
			# be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
			# will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
			# other authentication method such as basic auth or external auth useless - all requests will be allowed.
			#access_by_lua_block {
			#}
			
			header_filter_by_lua_block {
				lua_ingress.header()
				plugins.run()
			}
			
			body_filter_by_lua_block {
				plugins.run()
			}
			
			log_by_lua_block {
				balancer.log()
				
				monitor.call()
				
				plugins.run()
			}
			
			port_in_redirect off;
			
			set $balancer_ewma_score -1;
			set $proxy_upstream_name "test-sample-service-8443";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;
			
			set $pass_server_port    $server_port;
			
			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;
			
			set $proxy_alternative_upstream_name "";
			
			client_max_body_size                    1m;
			
			proxy_set_header Host                   $best_http_host;
			
			# Pass the extracted client certificate to the backend
			
			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;
			
			proxy_set_header                        Connection        $connection_upgrade;
			
			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;
			
			proxy_set_header X-Forwarded-For        $remote_addr;
			
			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;
			
			proxy_set_header X-Scheme               $pass_access_scheme;
			
			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
			
			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";
			
			# Custom headers to proxied server
			
			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;
			
			proxy_buffering                         off;
			proxy_buffer_size                       4k;
			proxy_buffers                           4 4k;
			
			proxy_max_temp_file_size                1024m;
			
			proxy_request_buffering                 on;
			proxy_http_version                      1.1;
			
			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;
			
			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;
			
			proxy_pass https://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}
	## end server gateway.am.wso2.com

Please refer to the log snippets below, extracted from the Ingress controller pod. These logs indicate that the changes deployed via the aforementioned Ingress object were successfully reflected to the ingress controller pod.

I0205 03:33:29.090764       7 status.go:304] "updating Ingress status" namespace="test" ingress="sample-ingress" currentValue=[{"ip":"192.168.49.2"}] newValue=[]
I0205 03:33:29.095521       7 event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"test", Name:"sample-ingress", UID:"47acf45f-a623-4a1d-99bb-507ef06803e1", APIVersion:"networking.k8s.io/v1", ResourceVersion:"843457", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

Furthermore, please refer to the log line below, extracted from the Ingress controller pod, which indicates that the request made via the following curl command is routed to the configured Kubernetes service ('sample-service') through the Ingress controller.

192.168.49.1 - - [05/Feb/2024:07:54:10 +0000] "GET /pizzashack/1.0.0/menu HTTP/2.0" 200 848 "-" "curl/7.81.0" 59 0.024 [test-sample-service-8443] [] 10.244.0.157:8443 848 0.024 200 381b4ca3206abc3bab19a6756cfa0cbc

Curl command: curl --cert public.crt --key client.key -X 'GET' 'https://gateway.am.wso2.com/pizzashack/1.0.0/menu' -H 'accept: application/json' -kv

It seems that the issue here is that the annotations configured in the Ingress object to send the client certificate to the upstream server are not being reflected in the Ingress controller pod.

  • Why is there no TLS section in your ingress when your request is a HTTPS request
    Please refer to the provided Ingress configuration (sample-ingress).

Hope you now have a better understanding of the issue. Please let us know if you need further clarification.

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

  • Any chance you can deploy httpbun.com image as a deployment and configure both a ClusterIP service for it as well as a ingress for it. Then curl it for the /get api and show all the related information
  • Please ensure TLS is configured and you curl over TLS

@sajith-madhusanka
Copy link
Author

Hi @longwuyuan

We have further investigated the issue and identified that the following annotation is mandatory to enable client-side authentication.

nginx.ingress.kubernetes.io/auth-tls-secret: test/client-ca-certs

Hence, we have modified the Ingress object as follows, including the aforementioned annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-ingress
  namespace : test
  annotations:
   nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
   nginx.ingress.kubernetes.io/auth-tls-secret: test/client-ca-certs
   nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
   nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - gateway.am.wso2.com
  rules:
  - host: gateway.am.wso2.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: sample-service
            port:
              number: 8443

We were able to observe that the client certificate is sent to the upstream server in a HTTP header (ssl-client-cert) after applying the above modification.


curl --cert public.crt --key client.key -X 'GET' 'https://gateway.am.wso2.com/pizzashack/1.0.0/menu'  -H 'accept: application/json'  -k
{
  "path": "/pizzashack/1.0.0/menu",
  "headers": {
    "host": "gateway.am.wso2.com",
    "ssl-client-cert": "-----BEGIN%20CERTIFICATE-----%0AMIIDezCCAmOgAwIBAgIEZ9bcdTANBgkqhkiG9w0BAQsFADBgMQswCQYDVQQGEwJM%0ASzELMAkGA1UECBMCV1MxCzAJBgNVBAcTAlNMMRIwEAYDVQQKEwlsb2NhbGhvc3Qx%0ADzANBgNVBAsTBlNhaml0aDESMBAGA1UEAxMJbG9jYWxob3N0MB4XDTI0MDEyOTA1%0AMTMwM1oXDTI0MDQyODA1MTMwM1owYDELMAkGA1UEBhMCTEsxCzAJBgNVBAgTAldT%0AMQswCQYDVQQHEwJTTDESMBAGA1UEChMJbG9jYWxob3N0MQ8wDQYDVQQLEwZTYWpp%0AdGgxEjAQBgNVBAMTCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC%0AAQoCggEBAKP1dRlB3vFlaZdLL%2F3uDLHyOLaq1bcC4y1FKkFsS3XQ3t335NhvpOys%0AaypS4RJG%2FFfMQts5Xz5TaCaCmPqGzDpp%2BNUnHTbTKNUMd2rtJ7JFVcPfVojau4p3%0AunUdL6JS%2FSKWwSwuXb8fAThX8LZgdZoan4vPKXQEYs8DNPmrBAB89EA1ydPGbXQX%0Ag247jkCgNfuPcBzFqdYBC4s%2FF%2BXBOxqrhKe8Qr3KuZugDWG610drQ%2B1fyv4BC9%2FX%0Apwmas35a58A%2FIJeFu6WxNc6hifC6q9%2FvntMkptK1nTB9ykD3No7kD718YI%2B6f%2FnH%0AcjXC0TL6xX45udQa%2FRpdy9dBnIE%2FtoECAwEAAaM9MDswHQYDVR0OBBYEFOrAvkbd%0AgMNf6f1j8lXuzVbKY4JKMBoGA1UdEQQTMBGHBApkAAGCCWxvY2FsaG9zdDANBgkq%0AhkiG9w0BAQsFAAOCAQEAmqKlXkXqaKGL3gKrJ%2Fe0VyCSoGtzPv%2BBkPThmwZ0iCm9%0AfV61JrPWaE%2FvYlMH38hvuoxhQDot5k24LPhV9OrM0IMX2E4hvix2mlkVGwkkRudz%0AOusuE88mL8EUfALOS150xkCoa2UuYZn7OD531gqczyxHYMqCetJRdCRy26Am4gKd%0AEdwKkGjtjEkgwy8YeMqLL2J01tiD3Q2zKpIySwi1Xy%2FhNaF2uc8Q6YfGaFg0FgQW%0Axw%2FAgHlASGzk9xTPYjZD%2B9f70P2bxQhvzSnZZvq5ToxYU%2F8lD7ieA9YDuqm6fCXn%0AxsMJLH%2BdnH%2BoLZhG8utRkvN4nz%2BhPfbqjgtOne5n6A%3D%3D%0A-----END%20CERTIFICATE-----%0A",
    "ssl-client-verify": "SUCCESS",
    "ssl-client-subject-dn": "CN=localhost,OU=Sajith,O=localhost,L=SL,ST=WS,C=LK",
    "ssl-client-issuer-dn": "CN=localhost,OU=Sajith,O=localhost,L=SL,ST=WS,C=LK",
    "x-request-id": "945d52258fdab794cee7a6b34a1d3427",
    "x-real-ip": "192.168.49.1",
    "x-forwarded-for": "192.168.49.1",
    "x-forwarded-host": "gateway.am.wso2.com",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "https",
    "x-forwarded-scheme": "https",
    "x-scheme": "https",
    "user-agent": "curl/7.81.0",
    "accept": "application/json"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "gateway.am.wso2.com",
  "ip": "192.168.49.1",
  "ips": [
    "192.168.49.1"
  ],
  "protocol": "https",
  "query": {},
  "subdomains": [
    "am",
    "gateway"
  ],
  "xhr": false,
  "os": {
    "hostname": "sample-deployment-c6fbc8fd8-bvmnp"
  },
  "connection": {
    "servername": false
  },
  "clientCertificate": {}

Thanks & Regards
Sajith M

@longwuyuan
Copy link
Contributor

thanks
/close

@k8s-ci-robot
Copy link
Contributor

@longwuyuan: Closing this issue.

In response to this:

thanks
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
[SIG Network] Ingress NGINX
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@longwuyuan @k8s-ci-robot @sajith-madhusanka