SSL errors with from-to-www-redirect (ERR_CERT_AUTHORITY_INVALID/MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT)

[VengefulAncient]

NGINX Ingress controller version: v.0.44.0

Kubernetes version (use kubectl version): v1.19.9-gke-1400

Environment:

• Cloud provider or hardware configuration: GKE
• Install tools: Helm v3
• Others: certificates are issued by cert-manager

What happened:

Essentially the same issue as in #5675 except we're trying to redirect from www. to naked domain and not the other way around. Trying to load https://www.example.com in Chrome produces NET::ERR_CERT_AUTHORITY_INVALID error.

In Firefox, it's MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT and clicking Accept the Risk and Continue proceeds to correctly follow redirects and load the site. Subsequent requests then also work fine.

What you expected to happen:

Redirect from www. to naked domain works without errors.

How to reproduce it:

Ingress is configured in line with what was suggested by aledbf: no backend for www. but www. domain is included in TLS hosts. According to several threads on the topic I've perused, that approach should work.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-front
  namespace: example
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($host = example.co.uk) {
          return 301 https://example.com$request_uri;
      }
spec:
  ingressClassName: nginx
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: example-front
            port:
              number: 80
  - host: example.co.uk
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: example-front
            port:
              number: 80
  tls:
  - hosts:
    - example.com
    - www.example.com
    - example.co.uk
    - www.example.co.uk
    secretName: example-front-tls

Anything else we need to know:

1. Ingress controller logs mention: the server www.example.com has SSL configured but the SSL certificate does not contains a CN for example.com. Redirects will not work for HTTPS to HTTPS. This is false, I've checked and the certificate does contain CN for example.com.

2. The logs have the same warning for https://www.example.co.uk (we own a .co.uk version of our domain just to rewrite it to .com) but navigating to https://www.example.co.uk works exactly as expected: there is a 308 redirecting to https://example.co.uk and then a 301 to https://example.com. Why is this not happening for https://www.example.com? (Note that adding the same 301 redirect if block from www.example.com to example.com does not fix the issue, nor should it be the solution.)

3. On the first request after opening private browsing, requesting www.example.com or http://www.example.com in Chrome or Firefox will proceed as follows:

   • first 308 from www. to naked domain
   • then 308 from http to https

   Immediately after though, requesting either of these in a new tab results in the same error again. Firefox network console shows nothing, but Chrome logs a 307 from http to https and then fails on the next request which asks for https://www.example.com. (Manually prefixing the URL with https:// always results in the error, even on the first request.) Notice how on the first attempt, www. to naked domain happened first, and the issue did not occur. This mirrors the behaviour described in 2043, which is apparently not fixed to this date.

4. The issue can be bypassed by adding www.example.com as a host to spec.rules. I don't consider this a valid configuration or a solution to this issue for several reasons:

   • Documentation states:

     For HTTPS to HTTPS redirects is mandatory the SSL Certificate defined in the Secret, located in the TLS section of Ingress, contains both FQDN in the common name of the certificate.

     This directive is fulfilled by my configuration. There is nothing about having to add the www. domain as a backend.

   • It defeats the point of from-to-www-redirect annotation

   • It complicates ingress Helm templates and bloats the resulting manifest

   • It does not actually result in a redirect (it merely handles requests on www. as another backend), as evidenced by the following warnings in the logs:

     • Already exists an Ingress with "www.example.com" hostname. Skipping creation of redirection from "www.example.com" to "example.com".
     • Already exists an Ingress with "example.com" hostname. Skipping creation of redirection from "example.com" to "www.example.com".

Based on these findings, it appears that there is a long-existing bug with this annotation. People are using workarounds such as a conditional rewrite block and they really shouldn't need to. It would be really amazing to have this actually fixed and put all the relevant discussions to rest once and for all.

I've also noticed that the annotation documentation claims:

If at some point a new Ingress is created with a host equal to one of the options (like domain.com) the annotation will be omitted.

How exactly will it be "omitted"? My cluster does have other Ingress objects with example.com as a host (without www. or from-to-www-redirect on them) but the generated nginx.conf still has redirect blocks for example.com location so it does not look like anything was "omitted":

		set_by_lua_block $redirect_to {
			local request_uri = ngx.var.request_uri
			if string.sub(request_uri, -1) == "/" then
			request_uri = string.sub(request_uri, 1, -2)
			end
			return string.format("%s://%s%s", ngx.var.scheme, "example.com", request_uri)
		}
		return 308 $redirect_to;

[tokers]

@VengefulAncient Would you like to output the result of openssl x509 with feeding your certificate? We need to get insight of the SANs and Common Name.

[VengefulAncient]

@VengefulAncient Would you like to output the result of openssl x509 with feeding your certificate? We need to get insight of the SANs and Common Name.

Sure, here it is (of course, example.com is my actual domain, www. are preserved though):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            <omitted>
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: May  2 01:40:20 2021 GMT
            Not After : Jul 31 01:40:20 2021 GMT
        Subject: CN=example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    <omitted>
                Exponent: <omitted>
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                <omitted>
            X509v3 Authority Key Identifier: 
                <omitted>
            Authority Information Access: 
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/
            X509v3 Subject Alternative Name: 
                DNS:example.co.uk, DNS:example.com, DNS:www.example.co.uk, DNS:www.example.com
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : <omitted>
                    Timestamp : May  2 02:40:19.216 2021 GMT
                    Extensions: none
                    Signature : <omitted>
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : <omitted>
                    Timestamp : May  2 02:40:20.219 2021 GMT
                    Extensions: none
                    Signature : <omitted>
    Signature Algorithm: sha256WithRSAEncryption
                         <omitted>

[k8s-triage-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[VengefulAncient]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[VengefulAncient]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[VengefulAncient]

/remove-lifecycle stale

[rikatz]

Can you update to a supported version? Wanted to look into this, but it seems you are using a really old one.

[VengefulAncient]

Thanks for looking into this! I will update to the latest version and report back ASAP.

[VengefulAncient]

Sorry this took quite a while.

Upgraded to latest controller (1.2.0) via latest helm chart (4.1.0). After testing (both Firefox and Chrome), it seems that the issue in the form I have described it is gone: I don't have www hosts under rules but I do under tls.hosts, and it works as expected. However:

• Controller logs still mention the server www.example.com has SSL configured but the SSL certificate does not contains a CN for example.com. Redirects will not work for HTTPS to HTTPS. Either there is still something wrong and I haven't found the conditions for it to resurface yet, or this error message is misleading.
• Chrome also still does the thing where the first attempt to access http://www.example.com will do a 308 from www to naked domain and then another 308 from http to https (expected order), and on all subsequent attempts there's first a 307 from http to https, and then a 308 from www to naked domain - which is the reversed order that used to break, but now doesn't.

So my question is, was there actually anything changed/fixed that addressed this issue? I've been following the updates closely but have not seen any such commits, to my best knowledge. I'm wary of marking this as resolved due to the above points. It sounds more like it got fixed accidentally and might regress at any point.

[VengefulAncient]

Never mind, the error is still happening. However, it is only happening with one of our subdomains (we have two in our staging cluster, one for each development branch, e.g. master.example.com and develop.example.com). Likewise, the SSL certificate error I mentioned in the previous comment only occurs with that subdomain. They are configured absolutely identically though, and use the same ingress controller, Ingress template, namespace, etc. The only difference I am aware of is that the subdomain with the error has one service deployed on it that doesn't exist on the error-free subdomain.

Any ideas on what I should be looking at to find out why this works correctly for one of them but not the other?

[longwuyuan]

Please try the latest release v1.2.0 and update status.

[longwuyuan]

/remove-kind bug

[VengefulAncient]

Please try the latest release v1.2.0 and update status.

Please don't pollute the thread with bureaucracy if you are not even going to read the messages or contribute to investigation. I have clearly indicated that I have already done this.

[VengefulAncient]

/kind bug

[longwuyuan]

Try to provided related info using the the recommended https://kubernetes.github.io/ingress-nginx/deploy/#gce-gke and maybe avoid the personal comments.

Seems like a complicated issue and you have used helm chart. Not sure if yo have tried the recommended manifest. So the data provided may not have clarity. The cert you showed has definitely expired in 2021 for example.

Provide data that has relevance like ;

• kubectl -n ingress-nginx get all -o wide
• kubectl -n <> get ing -A -o wide
• kubectl -n describe secret
• kubectl -n describe ing
• Your complete curl command in full with -v
• Your complete openssl command in full and output
• Logs of the controller pod
• Confirm that the cert is self-signed
• Any other info that helps

No data here is clearly indicating a bug

[VengefulAncient]

I reiterate, please stop polluting the thread with unhelpful comments. You are wasting everyone's time. This thread has been opened months ago and the certificate has obviously been renewed since. I'd like to hear from @rikatz who indicated he was interested in investigating this bug, and not read through obvious FAQ. I have already posted plenty of relevant information, and the fact that you cannot see that this is a bug only indicates that you have not taken the time to read through and understand the issue.

[iamNoah1]

/help
/priority important-longterm

[k8s-ci-robot]

@iamNoah1:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help
/priority important-longterm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.