proxy timeout annotations have no effect on nginx

[eagleusb]

NGINX Ingress controller version: 0.10.2 / quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.10.2

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare metal / On premise
• OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
• Kernel (e.g. uname -a): 4.9.0-5-amd64 Basic structure  #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux
• Install tools: kubeadm
• Others: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.10.2

What happened:

NGINX Ingress Controller v0.10.2 configuration doesn't reflect the proxy timeout annotations per Ingress.

This Ingress definition doesn't work as expected :

---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ing-manh-telnet-client
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy‑connect‑timeout: 30
    nginx.ingress.kubernetes.io/proxy‑read‑timeout: 1800
    nginx.ingress.kubernetes.io/proxy‑send‑timeout: 1800
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  tls:
    - hosts:
      - "manh-telnet.ls.domain.io"
      secretName: "tls-certs-domainio"
  rules:
    - host: "manh-telnet.ls.domain.io"
      http:
        paths:
        - path: "/"
          backend:
            serviceName: svc-manh-telnet-client
            servicePort: http

The actual vhost :

            # Custom headers to proxied server

            proxy_connect_timeout                   30s;
            proxy_send_timeout                      180s;
            proxy_read_timeout                      180s;

What you expected to happen:

The wanted vhost :

            # Custom headers to proxied server

            proxy_connect_timeout                   30s;
            proxy_send_timeout                      1800s;
            proxy_read_timeout                      1800s;

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

[garagatyi]

I have the same issue with quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17. Instead of custom timeouts nginx.conf contains default ones(60s) in the location part.

[garagatyi]

My particular test uses Ingress config (I needed timeouts only, but added other just for the test case):

``` - apiVersion: extensions/v1beta1 kind: Ingress metadata: name: che-ingress annotations: ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600" nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/upstream-fail-timeout: "30" nginx.ingress.kubernetes.io/add-base-url: "true" nginx.ingress.kubernetes.io/affinity: "cookie" spec: rules: - host: 192.168.99.100.nip.io http: paths: - backend: serviceName: che-host servicePort: 8080 ```

Which generates upstream:

    upstream che-che-host-8080 {
        # Load balance algorithm; empty for round robin, which is the default

        least_conn;

        keepalive 32;

        server 172.17.0.6:8080 max_fails=0 fail_timeout=0;

    }

And server:

    server {
        server_name 192.168.99.100.nip.io ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location / {

            set $proxy_upstream_name "che-che-host-8080";

            set $namespace      "che";
            set $ingress_name   "che-ingress3";
            set $service_name   "";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            proxy_set_header ssl-client-cert        "";
            proxy_set_header ssl-client-verify      "";
            proxy_set_header ssl-client-dn          "";

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://che-che-host-8080;

        }

    }

So looks like none of the annotations take effect.

[akaGelo]

You have an incorrect char "-" in your annotations.
In my configuration nginx.ingress.kubernetes.io/proxy-read-timeout is written and it works on the same version (0.10.2).

[eagleusb]

I don't get the point, annotations were tested one by one and - is an acceptable YAML value. Can you elaborate a bit more about an incorrect - in my annotations ?

[SaaldjorMike]

The hyphens are not the normal hyphens - but just look like them.

What @akaGelo is trying to say is that if you use your browsers search option and put in a - then some of them will not be highlighted. These are those hyphens which are not the correct ones.

[eagleusb]

Oh now that's seems very obvious ! Thanks guys, that was a pretty simple mistake. I'll look into official documentation if we can improve that with the same type of character for working copy/paste.

@garagatyi Maybe you have the same problem ? You should also update your ingress revision.

[garagatyi]

@gooodmorningopenstack I have older nginx controller, not wrong characters. The thing is I can't control the version of the controller, so I have to allow users to redefine controller annotations (it can be even not nginx controller at all). But thanks for your suggestion!

[mau21mau]

I'm having the same issue. Unlike the author one, my ingress doesn't have any special hyphen.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-routes
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-read-timeout: 1200
    nginx.ingress.kubernetes.io/proxy-send-timeout: 1200
spec:
  tls:
  - secretName: nginxsecret
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: frontend
          servicePort: 8000
      - path: /cron/*
        backend:
          serviceName: esg
          servicePort: 8000

      - path: /task/*
        backend:
          serviceName: esg
          servicePort: 8000

      - path: /api/connections/update/*
        backend:
          serviceName: esg
          servicePort: 8000

      - path: /api/drive/scansheet/*
        backend:
          serviceName: esg
          servicePort: 8000

[chris-mccoy]

I ran into this as well. I'm assuming an integer is required for timeouts? I was using "5m" because Nginx docs seemed to show that I could. Changed to 300 and things worked great after that.

[aledbf]

Closing. As @akaGelo commented you have an issue with the -. Is my fault. I am sure you copy/paste from the docs (a good thing) but in order to make readable the table the character was different
Please check #2111

[gae123]

I had the same problem and discovered that the following do not work:

nginx.ingress.kubernetes.io/proxy‑read‑timeout: 1800
nginx.ingress.kubernetes.io/proxy‑read‑timeout: 1800s
nginx.ingress.kubernetes.io/proxy‑read‑timeout: "1800s"

What does work is:

nginx.ingress.kubernetes.io/proxy‑read‑timeout: "1800"

[theRemix]

@gae123 that's not working for me

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  namespace: my-app
  annotations:
    nginx.org/websocket-services: "my-app"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "14400"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "14400"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "14400"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
spec: ...

still getting timed out after 30s

2018/03/31 16:55:07 Client 0xc420058b80 connected
2018/03/31 16:55:37 error: websocket: close 1006 (abnormal closure): unexpected EOF
2018/03/31 16:55:37 Client 0xc420058b80 disconnected

2018/03/31 16:58:19 Client 0xc420138e80 connected
2018/03/31 16:58:49 error: websocket: close 1006 (abnormal closure): unexpected EOF
2018/03/31 16:58:49 Client 0xc420138e80 disconnected

[aledbf]

kubernetes.io/ingress.class: "gce"

I seem you are using the GCE ingress controller. This annotation only works in nginx

[Tim-Schwalbe]

For me this is not working. Anyone sees an issue?
I am using the nginx helm chart: nginx-ingress-0.8.9

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: zalenium
  namespace: zalenium
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/auth-type: basic
    ingress.kubernetes.io/auth-secret: zalenium-basic-auth
    ingress.kubernetes.io/auth-realm: "Authentication Required"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "*"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
spec:
  rules:
  - host: "test.whatever"
    http:
      paths:
      - path: /
        backend:
          serviceName: zalenium
          servicePort: 4444