brotli causing errors in Safari for JavaScript files starting with 0.10.0

[gruberro]

Hi!

After having upgraded from 0.9.0 to 0.10.0 I recognized that Safari starts failing when loading JavaScript files (and yep, other file-types like CSS are working as expected 😟 ). The error message in Safari (and only Safari) is Failed to load resource: The operation couldn’t be completed. (kCFErrorDomainCFNetwork error 303.). I tried to find the root cause and ended up with disabling brotli. This helped to avoid the issue in Safari. A downgrade to 0.9.0 solved the errors as well (for sure with brotli enabled).

I tried to diff the generated nginx.confs and all other configuration files but couldn't find hints regarding this problem. One potential problem is discussed in google/ngx_brotli#50. Not pretty sure if the minor Nginx upgrade brings in the problem?!

Some details about the ingress configuration (non of them seems to be directly related to the problem):

• Google Cloud
• tls-acme on
• affinity and session-cookie-* on
• force-ssl-redirect on
• no special Nginx config settings given beside a small increase for the proxy-body-size and the ssl-protocols

Please let me know if I can support you with more details.
TY
Robert

[aledbf]

@gruberro what's the output using curl to make a request?

[gruberro]

This is the curl output you asked for. some.js is downloaded properly.

$ curl -v https://www.domain.com/static/app.76b6d4b4f4a513daa3d6.js > some.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 35.187.182.37...
* TCP_NODELAY set
* Connected to www.domain.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2482 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.domain.com
*  start date: Jan  9 21:51:59 2018 GMT
*  expire date: Apr  9 21:51:59 2018 GMT
*  subjectAltName: host "www.domain.com" matched cert's "www.domain.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f9fce000000)
> GET /static/app.76b6d4b4f4a513daa3d6.js HTTP/2
> Host: www.domain.com
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer:
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: nginx/1.13.8
< date: Tue, 23 Jan 2018 20:57:10 GMT
< content-type: application/javascript
< content-length: 1177555
< vary: Accept-Encoding
< set-cookie: _isc=a6492e1ff720125063583d2ec5cce3d6c119045a; Path=/; HttpOnly
< last-modified: Tue, 23 Jan 2018 16:27:47 GMT
< etag: "5a676283-11f7d3"
< expires: Wed, 23 Jan 2019 20:57:10 GMT
< cache-control: max-age=31536000
< x-frame-options: SAMEORIGIN
< accept-ranges: bytes
< strict-transport-security: max-age=15724800; includeSubDomains;
<
{ [3752 bytes data]
100 1149k  100 1149k    0     0  2091k      0 --:--:-- --:--:-- --:--:-- 2090k
* Connection #0 to host www.domain.com left intact

[day0ops]

I too was seeing a similar issue loading Grafana JS. But what I was seeing was net::ERR_SPDY_PROTOCOL_ERROR errors. Downgrading to 0.9.0 seem to be working so far.

[aledbf]

@gruberro @nixgadget I think I will change the default for the brotli module until we could determine exactly if the issue is in the brotli module or nginx 1.13.8

[aledbf]

@nixgadget can test if setting enable-brotli: "false" in the configuration configmap solves the issue?

[day0ops]

@aledbf can confirm that it appears to be working with 0.10.0 and enable-brotli: "false" set

[gruberro]

@aledbf thank you for investigating this problem. We're also going forward without brotli for the moment!

[cehoffman]

I have also seen this with javascript files and it is on Chrome 63.0.3239.132. Using 0.10.2 and enabling brotli can cause the error, so I'm pretty sure it is brotli at fault. In chrome on the javascript file when it fails, we get a similar net::ERR_SPDY_PROTOCOL_ERROR.

[aledbf]

For the next release, I will rollback the brotli version. I think the latest version is the reason for this error

[aledbf]

@gruberro @nixgadget please test quay.io/aledbf/nginx-ingress-controller:0.323 with brotli enabled

[gruberro]

@aledbf tested the container you mentioned and it's working now! TY

[day0ops]

@aledbf Works for me as well. Thanks a bunch for getting a fix quick.

[nottrobin]

What's the actual root of this? A bug in the Brotli algorithm? Is it being addressed?