Add check to make sure traefik can access https.tls.cert and https.tls.key

[deeduda]

Update by Erik 2021-10-25

I think the resolution to this issue was addressed by #462 (comment) suggesting a location where traefik could access the cert/key configured under https.tls.cert and https.tls.key.

Perhaps we could run a check to verify our traefik user will have access to the configured location, and if not, emit a error suggesting putting them in the /opt/tljh/state location?

Original issue

I have not been able to run TLJH in https mode due to traefik spilling out these errors.

The config is very simple and I am able to use it in http mode:

eladmin@AZLAPNSGEA01:~$ sudo tljh-config show
users:
  admin:
  - eladmin
https:
  tls:
    cert: /root/ssl/jupyter_platform_manulife_io_cert.cer
    key: /root/ssl/jupyter.platform.manulife.io.key
  enabled: false
auth:
  FirstUseAuthenticator:
    create_users: true

Both the cert and key are signed by a commercial provider and checked by openssl verify

I believed tljh is running 1.17 traefik.

Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback."
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Traefik version v1.7.18 built on 2019-09-26_01:56:30PM"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#c
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=error msg="failed to load X509 key pair: tls: failed to find any PEM data in certificate input"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Preparing server https &{Address::443 TLS:0xc000597950 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyPr
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=error msg="Unable to add a certificate to the entryPoint \"https\" : unable to generate TLS certificate : tls: failed to find any PEM data in certificate
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc00036c840 WhitelistSourceRange:[] WhiteList:0xc000863f00 Co
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc00063c440 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProt
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Starting server on :443"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Starting server on 127.0.0.1:8099"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Starting server on :80"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"rules.toml\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":0,\"Debug
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=error msg="failed to load X509 key pair: tls: failed to find any PEM data in certificate input"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Server configuration reloaded on :80"
Nov 08 18:11:47 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:47Z" level=info msg="Server configuration reloaded on :443"
Nov 08 18:11:49 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:49Z" level=error msg="failed to load X509 key pair: tls: failed to find any PEM data in certificate input"
Nov 08 18:11:49 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:49Z" level=info msg="Server configuration reloaded on :443"
Nov 08 18:11:49 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:49Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Nov 08 18:11:49 AZLAPNSGEA01 traefik[88964]: time="2019-11-08T18:11:49Z" level=info msg="Server configuration reloaded on :80"

[pulponair]

@deeduda you did not enable https:
enabled: false :)))

[deeduda]

Sorry I disable it again to get jupyterhub to work but it was enabled when I got these errors. Besides traefik won't attempt to use the certificate if https is disabled.

eladmin@AZLAPNSGEA01:~$ sudo tljh-config show
users:
  admin:
  - eladmin
https:
  tls:
    cert: /root/ssl/jupyter_platform_manulife_io_cert.cer
    key: /root/ssl/jupyter.platform.manulife.io.key
  enabled: true
auth:
  FirstUseAuthenticator:
    create_users: true

[pulponair]

@deeduda
Are you sure the cert is readable by traefik and the format is corrected ? Putting SSL certs to the home directory of root is a little odd btw. What does:
openssl x509 -in /root/ssl/jupyter_platform_manulife_io_cert.cer -text -noout
gives you?

[deeduda]

$ sudo openssl x509 -in /root/ssl/jupyter_platform_manulife_io_cert.cer -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            9b:89:24:8e:45:8d:98:38:3a:92:18:31:3a:ba:2e:59
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Organization Validation Secure Server CA
        Validity
            Not Before: Oct 29 00:00:00 2019 GMT
            Not After : Oct 28 23:59:59 2021 GMT
        Subject: C = CA, postalCode = M4W 1E5, ST = Ontario, L = Toronto, street = 200 Bloor Street East, O = Manulife Financial, OU = Global Infrastructure, OU = Enterprise SSL Pro, CN = jupyter.platform.manulife.io
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:96:93:f5:70:f3:9f:05:c1:86:b8:75:30:68:c1:
                    30:46:b2:f0:6e:a9:90:f2:2d:c6:f7:99:88:7e:ea:
                    f5:ee:73:0a:19:af:d4:70:8f:fa:4e:5d:a4:23:b8:
                    76:78:8a:2f:22:48:60:0b:0d:f2:d9:ca:4d:be:2d:
                    95:6e:ef:97:52:d4:56:cf:83:51:97:9f:df:b6:6c:
                    2e:fa:5b:87:3e:14:31:7d:59:3d:37:f0:e0:3a:37:
                    cd:17:95:cb:1d:67:24:62:26:9c:5b:bd:81:f2:5c:
                    84:5f:d4:7d:b8:fb:65:57:96:1c:b8:97:cd:a0:ba:
                    aa:70:99:a8:a3:5d:fc:a4:bf:20:15:83:90:6b:72:
                    3a:8b:99:ef:18:a2:12:71:81:a5:e2:fa:f1:65:20:
                    e6:7e:51:c2:0a:e9:ab:f1:7f:08:3f:86:3f:13:e1:
                    e8:6f:58:c4:2c:c2:8e:a7:be:56:1a:7f:84:97:85:
                    bf:31:6d:db:fc:dc:e5:08:2b:20:0c:71:b8:b3:ff:
                    cb:af:0b:e0:69:2c:53:0c:c6:5d:11:5c:88:dc:85:
                    ec:bf:e7:ef:d5:2b:ae:3d:74:f4:4c:0a:cc:7d:5a:
                    f6:42:a0:d4:0f:0b:8a:40:ed:3e:15:61:cd:b8:1f:
                    bf:23:85:99:43:55:4d:05:a2:88:d5:a3:57:f2:f6:
                    26:c1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:17:D9:D6:25:27:67:F9:31:C2:49:43:D9:30:36:44:8C:6C:A9:4F:EB

            X509v3 Subject Key Identifier:
                20:5C:09:9B:EC:1B:26:22:8B:2D:DE:EE:FE:7B:36:38:88:E3:E6:D6
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Certificate Policies:
                Policy: 1.3.6.1.4.1.6449.1.2.1.3.4
                  CPS: https://sectigo.com/CPS
                Policy: 2.23.140.1.2.2

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crl

            Authority Information Access:
                CA Issuers - URI:http://crt.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crt
                OCSP - URI:http://ocsp.sectigo.com

            X509v3 Subject Alternative Name:
                DNS:jupyter.platform.manulife.io, DNS:www.jupyter.platform.manulife.io
            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 7D:3E:F2:F8:8F:FF:88:55:68:24:C2:C0:CA:9E:52:89:
                                79:2B:C5:0E:78:09:7F:2E:6A:97:68:99:7E:22:F0:D7
                    Timestamp : Oct 29 00:36:25.413 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:C6:5C:70:31:A3:8E:AF:05:DE:38:85:
                                F0:B8:C4:AE:1D:8B:C8:06:05:F3:E4:A6:0A:30:75:B3:
                                78:94:61:51:42:02:20:68:C6:93:D8:1F:42:A2:F1:D2:
                                8F:09:3B:A6:48:9E:6E:61:E5:35:94:75:98:58:7F:EE:
                                AA:AA:C4:E4:89:CE:13
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 44:94:65:2E:B0:EE:CE:AF:C4:40:07:D8:A8:FE:28:C0:
                                DA:E6:82:BE:D8:CB:31:B5:3F:D3:33:96:B5:B6:81:A8
                    Timestamp : Oct 29 00:36:25.391 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:A7:D2:3B:38:14:40:5F:B3:02:05:3F:
                                76:82:31:61:57:A4:6C:22:EA:5E:8C:20:CB:94:5F:38:
                                32:48:56:5F:AB:02:21:00:D3:CE:78:34:55:DD:EF:39:
                                B4:E0:36:47:26:E0:11:A1:BE:43:C8:DA:3B:1E:89:C2:
                                31:45:4A:07:C3:1B:97:48
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 55:81:D4:C2:16:90:36:01:4A:EA:0B:9B:57:3C:53:F0:
                                C0:E4:38:78:70:25:08:17:2F:A3:AA:1D:07:13:D3:0C
                    Timestamp : Oct 29 00:36:25.352 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:44:A0:57:AA:C8:4E:56:2D:DE:12:00:8F:
                                15:2C:73:04:AD:B3:31:75:BA:77:4B:94:CF:80:EF:68:
                                E4:A8:64:0E:02:20:76:7A:26:A3:7A:FE:E0:D4:15:4F:
                                C4:10:AA:B6:58:E8:D4:AE:BB:54:21:20:16:5D:05:69:
                                14:31:05:DA:4F:06
    Signature Algorithm: sha256WithRSAEncryption
         64:08:b8:d0:ef:60:13:e9:cf:87:b6:30:d9:dc:0c:1a:cd:2b:
         cb:02:c5:15:10:1a:04:60:5f:0e:67:3a:4a:08:60:da:c9:fd:
         b2:3e:19:92:30:2c:0b:87:74:fc:c7:1e:c6:99:6b:9d:1d:54:
         26:9c:b4:5a:7f:8f:d0:1d:2c:f9:8d:ac:d0:1e:0b:97:f4:62:
         04:ad:1b:b1:74:6d:d6:1f:1d:78:ab:7a:0d:be:46:30:b0:6e:
         66:f9:d4:06:c3:22:9b:cb:f0:09:17:3d:5f:81:6c:2b:4f:5c:
         d0:27:b3:db:de:1e:72:75:ac:63:63:8f:1d:12:ef:fb:e3:94:
         52:e8:4b:57:e5:e5:43:27:0c:b2:a5:89:d6:4a:3f:83:2f:d0:
         ab:f3:c1:d6:9c:25:e3:a6:6d:b5:26:34:aa:39:13:9b:83:59:
         2e:a2:8e:b6:bc:20:d7:a6:89:6b:d8:93:70:0f:d3:dc:73:dd:
         44:9e:0d:ae:c5:5d:7f:36:aa:35:7c:cb:2e:af:92:d3:ae:6f:
         92:8c:2f:46:d2:ab:e7:23:0b:9c:fc:88:3a:2d:a8:51:ef:72:
         1c:19:81:7f:74:90:43:6d:aa:57:3e:6e:e6:4b:12:50:d8:4e:
         6a:97:66:9e:81:20:78:9e:49:2a:f2:9a:66:ae:1b:ba:da:f9:
         c5:4a:6a:84

[deeduda]

I tried to move it to /tmp and other directory and the same thing happened.

The odd thing is I can trace the go lang to this particular function and it seems to be looking for a default certificate in the store.

https://github.com/containous/traefik/blob/5a3e3257422ab3c29dc9e44fd5a834ec8bd25fcf/pkg/tls/tlsmanager.go

look for ""failed to load X509 key pair"

[GeorgianaElena]

Hey @deeduda! Sorry for the late reply.
Try putting the cert and key in /opt/tljh/state. This is the working directory of traefik and should have access to it.
Hope it helps.

[mauro3]

I ran into this as well. Putting the cert & key into /opt/tljh/state worked for me. Tnx