-
Notifications
You must be signed in to change notification settings - Fork 8.5k
Description
What happened:
Reopening issue which is the same as #4832
I defined an ingress resource with a server alias on a separate domain, using the nginx.ingress.kubernetes.io/server-alias, and 2 certificates one wildcard that matches the primary domain and a wildcard host that matches the alias. When sending a request that matches the alias but not the primary host, the fake self-signed certificate is used. When sending a request that matches the primary host, the configured certificate is used. If I manually specify a different subdomain on the server-alias
as a host in the ingress the certificate is loaded as intended for the service-alias
subdomain.
What you expected to happen:
I expected to receive the configured certificate for the server-alias
used in the ingress, without having to define it as a host.
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
NGINX Ingress controller
Release: v1.10.0
Build: 71f78d4
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.3
Kubernetes version (use kubectl version
):
Client Version: v1.28.7+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7+k3s1
Environment:
-
Cloud provider or hardware configuration:
-
OS (e.g. from /etc/os-release): Fedora 39
-
Kernel (e.g.
uname -a
): Linux fedora 6.6.13-200.fc39.x86_64 Basic structure #1 SMP PREEMPT_DYNAMIC Sat Jan 20 18:03:28 UTC 2024 x86_64 GNU/Linux -
Install tools:
- k3s
-
Basic cluster related info:
- v1.28.7+k3s1
- fedora Ready control-plane,master 158m v1.28.7+k3s1 192.168.1.109 Fedora Linux 39 (Workstation Edition) 6.6.13-200.fc39.x86_64 containerd://1.7.11-k3s2
-
How was the ingress-nginx-controller installed:
- If helm was used then please show output of
helm ls -A | grep -i ingress
ingress-nginx ingress-nginx 1 2024-03-04 13:59:07.9611125 -0800 PST deployed ingress-nginx-4.10.0 1.10.0
- If helm was used then please show output of
helm -n <ingresscontrollernamespace> get values <helmreleasename>
- If helm was used then please show output of
USER-SUPPLIED VALUES:
null
-
If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
-
if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
-
Current State of the controller:
kubectl describe ingressclasses
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
Controller: k8s.io/ingress-nginx
Events: <none>
kubectl -n <ingresscontrollernamespace> get all -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system pod/coredns-6799fbcd5-l5f94 1/1 Running 1 (109m ago) 170m 10.42.0.5 fedora <none> <none>
kube-system pod/local-path-provisioner-6c86858495-9vsg8 1/1 Running 1 (109m ago) 170m 10.42.0.6 fedora <none> <none>
kube-system pod/svclb-ingress-nginx-controller-c11be5c2-zpz96 2/2 Running 2 (109m ago) 141m 10.42.0.4 fedora <none> <none>
default pod/demo-5f7bb54887-qkhmf 1/1 Running 1 (109m ago) 140m 10.42.0.2 fedora <none> <none>
ingress-nginx pod/ingress-nginx-controller-6dc8c8fdf4-jwwbp 1/1 Running 1 (109m ago) 141m 10.42.0.3 fedora <none> <none>
kube-system pod/metrics-server-67c658944b-6gjkg 1/1 Running 1 (109m ago) 170m 10.42.0.7 fedora <none> <none>
default pod/echo-test-864d879bcf-xl2qf 1/1 Running 0 64m 10.42.0.8 fedora <none> <none>
default pod/echo-test-864d879bcf-96bfj 1/1 Running 0 64m 10.42.0.9 fedora <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 170m <none>
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 170m k8s-app=kube-dns
kube-system service/metrics-server ClusterIP 10.43.85.57 <none> 443/TCP 170m k8s-app=metrics-server
ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.43.51.35 <none> 443/TCP 141m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
default service/demo ClusterIP 10.43.129.220 <none> 80/TCP 140m app=demo
ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.28.240 192.168.1.109 80:32206/TCP,443:31521/TCP 141m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
default service/echo-test ClusterIP 10.43.24.96 <none> 80/TCP 64m app=echo-test
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
kube-system daemonset.apps/svclb-ingress-nginx-controller-c11be5c2 1 1 1 1 1 <none> 141m lb-tcp-80,lb-tcp-443 rancher/klipper-lb:v0.4.5,rancher/klipper-lb:v0.4.5 app=svclb-ingress-nginx-controller-c11be5c2
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
kube-system deployment.apps/local-path-provisioner 1/1 1 1 170m local-path-provisioner rancher/local-path-provisioner:v0.0.26 app=local-path-provisioner
kube-system deployment.apps/coredns 1/1 1 1 170m coredns rancher/mirrored-coredns-coredns:1.10.1 k8s-app=kube-dns
default deployment.apps/demo 1/1 1 1 140m nginx nginx app=demo
ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 141m controller registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system deployment.apps/metrics-server 1/1 1 1 170m metrics-server rancher/mirrored-metrics-server:v0.6.3 k8s-app=metrics-server
default deployment.apps/echo-test 2/2 2 2 64m echo-test nginxdemos/hello app=echo-test
NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
kube-system replicaset.apps/local-path-provisioner-6c86858495 1 1 1 170m local-path-provisioner rancher/local-path-provisioner:v0.0.26 app=local-path-provisioner,pod-template-hash=6c86858495
kube-system replicaset.apps/coredns-6799fbcd5 1 1 1 170m coredns rancher/mirrored-coredns-coredns:1.10.1 k8s-app=kube-dns,pod-template-hash=6799fbcd5
default replicaset.apps/demo-5f7bb54887 1 1 1 140m nginx nginx app=demo,pod-template-hash=5f7bb54887
ingress-nginx replicaset.apps/ingress-nginx-controller-6dc8c8fdf4 1 1 1 141m controller registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6dc8c8fdf4
kube-system replicaset.apps/metrics-server-67c658944b 1 1 1 170m metrics-server rancher/mirrored-metrics-server:v0.6.3 k8s-app=metrics-server,pod-template-hash=67c658944b
default replicaset.apps/echo-test-864d879bcf 2 2 2 64m echo-test nginxdemos/hello app=echo-test,pod-template-hash=864d879bcf
kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
Name: ingress-nginx-controller-6dc8c8fdf4-jwwbp
Namespace: ingress-nginx
Priority: 0
Service Account: ingress-nginx
Node: fedora/192.168.1.109
Start Time: Mon, 04 Mar 2024 13:59:18 -0800
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
pod-template-hash=6dc8c8fdf4
Annotations: <none>
Status: Running
IP: 10.42.0.3
IPs:
IP: 10.42.0.3
Controlled By: ReplicaSet/ingress-nginx-controller-6dc8c8fdf4
Containers:
controller:
Container ID: containerd://7bb9af2efeeef87b0e9afa6a3cb57a508159bef066a8f034eed5ad056126bc4c
Image: registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
Image ID: registry.k8s.io/ingress-nginx/controller@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
SeccompProfile: RuntimeDefault
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-nginx-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
--enable-metrics=false
State: Running
Started: Mon, 04 Mar 2024 14:31:28 -0800
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Mon, 04 Mar 2024 13:59:30 -0800
Finished: Mon, 04 Mar 2024 14:31:26 -0800
Ready: True
Restart Count: 1
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6dc8c8fdf4-jwwbp (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hv6n6 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-hv6n6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RELOAD 30m (x8 over 110m) nginx-ingress-controller NGINX reload triggered due to a change in configuration
kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.28.240
IPs: 10.43.28.240
LoadBalancer Ingress: 192.168.1.109
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32206/TCP
Endpoints: 10.42.0.3:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31521/TCP
Endpoints: 10.42.0.3:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
-
Current state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o wide
kubectl -n <appnamespace> describe ing <ingressname>
- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
-
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
kubectl describe ...
of any custom configmap(s) created and in use- Any other related information that may help
- Any other related information like ;
How to reproduce this issue:
Install K3s (Any Distro, Seen on EKS as well)
- K3s https://docs.k3s.io/installation
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh -s -
Install the ingress controller using helm
helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace
Create Dummy certificates
# Create a root CA
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -subj "/C=US/ST=New York/L=New York/O=Example Inc./OU=Root CA/CN=example.com"
cat > cluster_foo_example.conf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn
[dn]
C = US
ST = New York
L = New York
O = "Example, Inc."
OU = "Cluster Foo Example"
CN = cluster.foo.example
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = cluster.foo.example
DNS.2 = *.cluster.foo.example
EOF
openssl req -new -nodes -x509 -newkey rsa:2048 -keyout cluster_foo_example.key -out cluster_foo_example.crt -config cluster_foo_example.conf -days 365
cat > random_bar_example.conf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
x509_extensions = v3_req
distinguished_name = dn
[dn]
C = US
ST = California
L = San Francisco
O = "Bar Inc."
OU = "Random Bar Example"
CN = random.bar.example
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = random.bar.example
DNS.2 = *.random.bar.example
EOF
openssl req -new -nodes -x509 -newkey rsa:2048 -keyout random_bar_example.key -out random_bar_example.crt -config random_bar_example.conf -days 365
# Create the Kubernetes secret for cluster.foo.example
kubectl create secret tls cluster-foo-tls \
--key cluster_foo_example.key \
--cert cluster_foo_example.crt \
--namespace ingress-nginx
# Create the Kubernetes secret for random.bar.example
kubectl create secret tls random-bar-tls \
--key random_bar_example.key \
--cert random_bar_example.crt \
--namespace ingress-nginx
Install an application that will act as default backend (is just an echo app)
kubectl apply -n ingress-nginx -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml
Create an ingress (please add any additional annotation required)
echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/server-alias: 'test.cluster.foo.example'
name: test-ingress
namespace: ingress-nginx
spec:
ingressClassName: nginx
tls:
- hosts:
- cluster.foo.example
- '*.cluster.foo.example'
secretName: cluster-foo-tls
- hosts:
- random.bar.example
- '*.random.bar.example'
secretName: random-bar-tls
rules:
- host: test.random.bar.example
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80
" | kubectl apply -f -
make a request
Update /etc/host
to point to 127.0.0.1 for test.cluster.foo.example
and test.random.bar.example
➜ curl --insecure -vvI https://test.random.bar.example 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Bar Inc.; OU=Random Bar Example; CN=random.bar.example
➜ curl --insecure -vvI https://test.cluster.foo.example/ 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Mar 5 06:54:48 2024 GMT
* expire date: Mar 5 06:54:48 2025 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
Expect to see test.cluster.foo.example
using the correct certificate
Anything else we need to know: If you add a valid host entry the cert is loaded (see comment in manifest above)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status