TLS authentication ignored by nginx ingress

[christian-roggia]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): verify-client, auth-tls-secret, auth-tls

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.9.0-beta.15 (with RBAC)

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare metal (kubeadm + Flannel)
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a): Linux 4.4.0-81-generic

What happened:
Kubernetes dashboard and Grafana dashboard are accessible from everywhere without a valid CA.

What you expected to happen:
I expected an error message because I did not have a valid CA configured in Chrome.
I expected an error message 503 if my configuration was not correct.

How to reproduce it (as minimally and precisely as possible):
Follow the example provided by the docs:
https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/auth/client-certs/nginx-tls-auth.yaml

Anything else we need to know:
I generated the CA and TLS secrets by following the guide provided by the docs:
https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md
I only changed the name and namespace.

NOTE: Basic-auth works correctly with my kubernetes and nginx ingress configuration.
NOTE: If I try to invalidate the configuration of the TLS auth by setting an invalid CA the error page 503 is displayed as expected.

---

I am currently using the following yaml configuration to deploy my ingress (same config for kubernetes' dashboard):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/auth-tls-secret: "kube-system/dashboard-ca-ingress"
    ingress.kubernetes.io/auth-tls-verify-depth: "3"
    ingress.kubernetes.io/auth-tls-verify-client: "on"
    ingress.kubernetes.io/force-ssl-redirect: "true"
  name: grafana-ingress
  namespace: kube-system
spec:
  tls:
  - hosts:
    - <host goes here>
    secretName: nginx-tls-secret
  rules:
  - host: <host goes here>
    http:
      paths:
      - path: /
        backend:
          serviceName: monitoring-grafana
          servicePort: 80

dashboard-ca-ingress secret:

apiVersion: v1
data:
  ca.crt: <data goes here>
kind: Secret
metadata:
  name: dashboard-ca-ingress
  namespace: kube-system
type: Opaque

nginx-tls-secret secret:

apiVersion: v1
data:
  tls.crt: <data goes here>
  tls.key: <data goes here>
kind: Secret
metadata:
  name: nginx-tls-secret
  namespace: kube-system
type: kubernetes.io/tls

[christian-roggia]

If someone could run my ingress in a kubernetes cluster (AWS, GKE, etc.) where the dashboard and heapster are installed and verify whether there is something wrong with my cluster or if the TLS authentication is indeed ignored would be extremely helpful. I am stuck with this problem and I would like to know if it is a bug or if I missed something while I was configuring the nignx ingress.

[rikatz]

@christian-roggia Can you please provide in some gist the nginx.conf generated by Ingress controller? To do so, exec a kubectl exec -n <ingress-namespace> <POD> cat /etc/nginx/nginx.conf

Please configure CA Certificate first, so we can figure out if the file is being generated correctly

Thanks

[christian-roggia]

Here is the output of kubectl describe ingress grafana-ingress -n kube-system:

Name:             grafana-ingress
Namespace:        kube-system
Address:          <address goes here>
Default backend:  default-http-backend:80 (<none>)
TLS:
  nginx-tls-secret terminates <domain goes here>
Rules:
  Host             Path  Backends
  ----             ----  --------
  grafana.weeb.io
                   /   monitoring-grafana:80 (<clusterip goes here>:3000)
Annotations:
  auth-tls-verify-depth:   3
  force-ssl-redirect:      true
  auth-tls-secret:         kube-system/dashboard-ca-ingress
  auth-tls-verify-client:  on
Events:                    <none>

here is the output of kubectl exec -n <ingress-namespace> <POD> cat /etc/nginx/nginx.conf:

daemon off;

worker_processes 1;
pid /run/nginx.pid;

worker_rlimit_nofile 1047552;
worker_shutdown_timeout 10s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;

    real_ip_recursive   on;
    set_real_ip_from    0.0.0.0/0;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    sendfile            on;

    aio                 threads;
    aio_write           on;

    tcp_nopush          on;
    tcp_nodelay         on;

    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout  75s;
    keepalive_requests 100;

    client_header_buffer_size       1k;
    client_header_timeout           60s;
    large_client_header_buffers     4 8k;
    client_body_buffer_size         8k;
    client_body_timeout             60s;

    http2_max_field_size            4k;
    http2_max_header_size           16k;

    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;

    proxy_headers_hash_max_size     512;
    proxy_headers_hash_bucket_size  64;

    variables_hash_bucket_size      64;
    variables_hash_max_size         2048;

    underscores_in_headers          off;
    ignore_invalid_headers          on;
    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    # Custom headers for response

    server_tokens on;

    # disable warnings
    uninitialized_variable_warn off;

    # Additional available variables:
    # $namespace
    # $ingress_name
    # $service_name
    log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver <address goes here> valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    map $http_x_forwarded_for $the_real_ip {
        default          $remote_addr;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
        default           $http_x_forwarded_port;
        ''                $server_port;
    }

    map $http_x_forwarded_host $best_http_host {
        default          $http_x_forwarded_host;
        ''               $this_host;
    }

    map $pass_server_port $pass_port {
        443              443;
        default          $pass_server_port;
    }

    # Obtain best http host
    map $http_host $this_host {
        default          $http_host;
        ''               $host;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    ssl_ecdh_curve auto;

    proxy_ssl_session_reuse on;

    upstream kube-system-kubernetes-dashboard-80 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;

        keepalive 32;

        server <clusterip goes here>:9090 max_fails=0 fail_timeout=0;
    }

    upstream kube-system-monitoring-grafana-80 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;

        keepalive 32;

        server <clusterip goes here>:3000 max_fails=0 fail_timeout=0;
    }

    upstream upstream-default-backend {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;

        keepalive 32;

        server <clusterip goes here>:8080 max_fails=0 fail_timeout=0;
    }
    server {
        server_name _;
        listen 80 default_server reuseport backlog=511;
        listen [::]:80 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        listen 443  default_server reuseport backlog=511 ssl http2;
        listen [::]:443  default_server reuseport backlog=511 ssl http2;
        # PEM sha: f0cf1173f476eea067b9e531742e9edf1f5d7aa9
        ssl_certificate                         /ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/default-fake-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

        location / {

            set $proxy_upstream_name "upstream-default-backend";

            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://upstream-default-backend;
        }

        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }
    server {
        server_name <domain goes here>;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
        # PEM sha: <sha goes here>
        ssl_client_certificate                  /ingress-controller/ssl/ca-kube-system-dashboard-ca-ingress.pem;
        ssl_verify_client                       on;
        ssl_verify_depth                        3;

        location / {

            set $proxy_upstream_name "kube-system-kubernetes-dashboard-80";

            set $namespace      "kube-system";
            set $ingress_name   "dashboard-ingress";
            set $service_name   "kubernetes-dashboard";

            # enforce ssl on server side
            if ($pass_access_scheme = http) {
                return 301 https://$best_http_host$request_uri;
            }
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
            # Pass the extracted client certificate to the backend
            proxy_set_header ssl-client-cert        $ssl_client_cert;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://kube-system-kubernetes-dashboard-80;
        }
    }
    server {
        server_name <domain goes here>;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
        # PEM sha: <sha goes here>
        ssl_client_certificate                  /ingress-controller/ssl/ca-kube-system-dashboard-ca-ingress.pem;
        ssl_verify_client                       on;
        ssl_verify_depth                        3;

        location / {

            set $proxy_upstream_name "kube-system-monitoring-grafana-80";

            set $namespace      "kube-system";
            set $ingress_name   "grafana-ingress";
            set $service_name   "monitoring-grafana";

            # enforce ssl on server side
            if ($pass_access_scheme = http) {
                return 301 https://$best_http_host$request_uri;
            }
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;
            # Pass the extracted client certificate to the backend
            proxy_set_header ssl-client-cert        $ssl_client_cert;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_real_ip;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://kube-system-monitoring-grafana-80;
        }
    }
    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/ingress/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
        listen 18080 default_server reuseport backlog=511;
        listen [::]:18080 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        location /healthz {
            access_log off;
            return 200;
        }

        location /nginx_status {
            set $proxy_upstream_name "internal";

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass          http://upstream-default-backend;
        }
    }
}

stream {
    log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # TCP services

    # UDP services
}

here is the output of here is the output of kubectl logs <POD> -n <ingress namespace> whenever i hit the grafana monitoring dashboard (sample):

W1023 23:26:22.196606       5 controller.go:1054] ssl certificate kube-system/nginx-tls-secret does not contain a Common Name or Subject Alternative Name for host <domain goes here>
W1023 23:26:22.196645       5 controller.go:1054] ssl certificate kube-system/nginx-tls-secret does not contain a Common Name or Subject Alternative Name for host <domain goes here>
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:37 +0000] "GET / HTTP/1.1" 200 2358 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 663 0.009 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 9802 0.009 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:37 +0000] "GET /public/css/fonts.min.css HTTP/1.1" 304 0 "https://<domain goes here>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 675 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:37 +0000] "GET /public/css/grafana.dark.min.af13213c.css HTTP/1.1" 304 0 "https://<domain goes here>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 691 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:37 +0000] "GET /public/app/boot.07485e0c.js HTTP/1.1" 304 0 "https://<domain goes here>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 663 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/dashboards/home HTTP/1.1" 200 645 "https://<domain goes here>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 656 0.006 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 1452 0.006 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/img/grafana_icon.svg HTTP/1.1" 304 0 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 707 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/img/icn-dashboard-tiny.svg HTTP/1.1" 304 0 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 713 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/1.1" 304 0 "https://<domain goes here>/public/css/grafana.dark.min.af13213c.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 755 0.000 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.000 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/fonts/grafana-icons.ttf?okx5td HTTP/1.1" 304 0 "https://<domain goes here>/public/css/grafana.dark.min.af13213c.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 746 0.000 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.000 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/fonts/opensans/cJZKeOuBrn4kERxqtaUH3VtXRa8TVwTICgirnJhmVJw.woff2 HTTP/1.1" 304 0 "https://<domain goes here>/public/css/fonts.min.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 764 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/search?limit=1 HTTP/1.1" 200 92 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 663 0.005 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 92 0.005 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/plugins?core=0&embedded=0 HTTP/1.1" 200 2 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 674 0.004 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 2 0.004 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /public/fonts/opensans/MTP_ySUJH_bn48VBG8sNSugdm0LZdjqr5-oayXSOefg.woff2 HTTP/1.1" 304 0 "https://<domain goes here>/public/css/fonts.min.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 764 0.001 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 0 0.001 304
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/search?limit=4 HTTP/1.1" 200 177 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 663 0.004 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 177 0.004 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/search?limit=4&starred=true HTTP/1.1" 200 2 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 676 0.004 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 2 0.004 200
<ip goes here> - [<ip goes here>] - - [23/Oct/2017:23:31:38 +0000] "GET /api/org/users HTTP/1.1" 200 81 "https://<domain goes here>/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36" 658 0.003 [kube-system-monitoring-grafana-80] 10.244.2.6:3000 81 0.003 200

[rikatz]

OK, by looking at this error:

W1023 23:26:22.196645       5 controller.go:1054] ssl certificate kube-system/nginx-tls-secret does not contain a Common Name or Subject Alternative Name for host <domain goes here>

It seems the certificate you're using in vHost does not contain the commonName used in vHost configuration (like grafana.domain.com). So Ingress refuses to use it :)

And looking at your conf, those vhosts doesn't contain a listen 443, and ssl certificate, probably because they couldn't look at their certificate.

Try creating a certificate to this vhosts with the right commonName, or removing the line secretName: nginx-tls-secret to use the default self generated one.

[christian-roggia]

Thank you @rikatz for your observation, I fixed the TLS certificate and the warning is gone, unfortunately this was completely unrelated to the issue:

server {
	server_name <domain goes here>;
	listen 80;
	listen [::]:80;
	set $proxy_upstream_name "-";

	listen 443  ssl http2;
	listen [::]:443  ssl http2;
	# PEM sha: <sha goes here>
	ssl_certificate                         /ingress-controller/ssl/kube-system-nginx-tls-dashboard.pem;
	ssl_certificate_key                     /ingress-controller/ssl/kube-system-nginx-tls-dashboard.pem;
	ssl_trusted_certificate                 /ingress-controller/ssl/kube-system-nginx-tls-dashboard-full-chain.pem;
	ssl_stapling                            on;
	ssl_stapling_verify                     on;

	more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
	# PEM sha: <sha goes here>
	ssl_client_certificate                  /ingress-controller/ssl/ca-kube-system-dashboard-ca-ingress.pem;
	ssl_verify_client                       on;
	ssl_verify_depth                        3;

	location / {

		set $proxy_upstream_name "kube-system-kubernetes-dashboard-80";

		set $namespace      "kube-system";
		set $ingress_name   "dashboard-ingress";
		set $service_name   "kubernetes-dashboard";

		# enforce ssl on server side
		if ($pass_access_scheme = http) {
			return 301 https://$best_http_host$request_uri;
		}
		port_in_redirect off;

		client_max_body_size                    "1m";

		proxy_set_header Host                   $best_http_host;
		# Pass the extracted client certificate to the backend
		proxy_set_header ssl-client-cert        $ssl_client_cert;

		# Allow websocket connections
		proxy_set_header                        Upgrade           $http_upgrade;
		proxy_set_header                        Connection        $connection_upgrade;

		proxy_set_header X-Real-IP              $the_real_ip;
		proxy_set_header X-Forwarded-For        $the_real_ip;
		proxy_set_header X-Forwarded-Host       $best_http_host;
		proxy_set_header X-Forwarded-Port       $pass_port;
		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
		proxy_set_header X-Original-URI         $request_uri;
		proxy_set_header X-Scheme               $pass_access_scheme;

		# Pass the original X-Forwarded-For
		proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

		# mitigate HTTPoxy Vulnerability
		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
		proxy_set_header Proxy                  "";

		# Custom headers to proxied server

		proxy_connect_timeout                   5s;
		proxy_send_timeout                      60s;
		proxy_read_timeout                      60s;

		proxy_redirect                          off;
		proxy_buffering                         off;
		proxy_buffer_size                       "4k";
		proxy_buffers                           4 "4k";
		proxy_request_buffering                 "on";

		proxy_http_version                      1.1;

		proxy_cookie_domain                     off;
		proxy_cookie_path                       off;

		# In case of errors try the next upstream server before returning an error
		proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

		proxy_pass http://kube-system-kubernetes-dashboard-80;
	}
}

I noticed that the verification of the variable $ssl_client_verify is missing in the nginx.conf:

if ($ssl_client_verify = FAILED) {
    return 495;
}

if ($ssl_client_verify = NONE) {
    return 496;
}

if ($ssl_client_verify != SUCCESS) {
    return 403;
}

Shouldn't the result of the CA authentication be verified?

According to the official documentation:

http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client

ssl_verify_client
Enables verification of client certificates. The verification result is stored in the $ssl_client_verify variable.

$ssl_client_verify
returns the result of client certificate verification: “SUCCESS”, “FAILED:reason”, and “NONE” if a certificate was not present;

[christian-roggia]

I verified now that if I disable Cloudflare's HTTP Proxy (CDN) the result is as expected:

400 Bad Request
No required SSL certificate was sent

However if CF proxy is turned on the server ignores the authentication and exposes the service to the internet.

[rikatz]

So you're calling through Cloudflare instead of directly calling Ingress right?

Isn't this the case to enable the Force of SSL Redirect?

[christian-roggia]

Hi @rikatz, thank you again for your feedback, SSL redirection is already enabled on both Cloudflare and ingress:

ingress.kubernetes.io/force-ssl-redirect: "true"

I verified with curl requests to a non-proxied domain:

• no certificate: "No required SSL certificate was sent" as expected
• invalid certificate: "The SSL certificate error" as expected
• valid certificate: 200 OK, as expected

Everything is indeed as expected. But as soon as I turn on the Cloudflare's HTTP Proxy (CDN) the services get exposed to the internet with no authentication required.

Cloudflare is bypassing the CA authentication.

[aledbf]

@christian-roggia please test the image quay.io/aledbf/nginx-ingress-controller:0.265

[christian-roggia]

Requests to Cloudflare return a 403 error page as expected:

403 Forbidden

Requests to non-proxied domain with a valid certificate return a 400 error page with no additional information:

< HTTP/1.1 400 Bad Request
< Server: nginx/1.13.6
< Date: Wed, 25 Oct 2017 00:13:52 GMT
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
<
* Connection #0 to host <domain> left intact
400 Bad Request

[christian-roggia]

I switched to basic authentication, it seems that Cloudflare only provides TLS with Client Authentication to Enterprise customers (and they do that on their own infrastructure not on the origin server):
https://blog.cloudflare.com/introducing-tls-client-auth/

Cloudflare only allows Authenticated Origin Pulls and they are providing their own certificate for that purpose:
https://blog.cloudflare.com/protecting-the-origin-with-tls-authenticated-origin-pulls/

I suggest to add to the documentation that TLS with Client Authentication is NOT possible and SHOULDN'T be enabled while using services like Cloudflare as they do not allow it and might result in unexpected behavior.

It would also be useful to point out that only Authenticated Origin Pulls are allowed and can be configured by following their tutorial:
https://support.cloudflare.com/hc/en-us/articles/204494148-Setting-up-NGINX-to-use-TLS-Authenticated-Origin-Pulls