Too many certificate files being generated for EKS Optimized AMI v.12

[zealvora]

What happened:

Server Certificates being generated every second in the /var/lib/kubelet/pki directory

Example Output:

-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-38.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-39.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-40.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-41.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-42.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-43.pem
-rw------- 1 root root 1232 Apr 10 08:23 kubelet-server-2019-04-10-08-23-44.pem

What you expected to happen: Limited number of PEM files.

How to reproduce it (as minimally and precisely as possible):

Launch Worker nodes from EKS Optimized AMI 1.12
Associate them with EKS Cluster

Environment:

• AWS Region: us-east-1
• Instance Type(s): t2.medium
• AMI Version: EKS Optimized AMI v1.12

[janlau1111]

We also encounter this problem. amazon-eks-node-1.12-v20190329 (ami-08716b70cac884aaa)

[micahhausler]

Hi, thanks for reporting this. This issue affects 1.12 and is a result of the EKS certificate signer not adding public DNS names and public IPs to kubelet certificates, causing the kubelet to re-request a new certificate ever few seconds.

We'll be rolling out a control plane fix to resolve this issue for 1.12 clusters shortly. If this is impacting kubelet operations for you, as a mitigation you can either use the 1.11 AMI, or run your nodes in non-public subnets and you should see this behavior stop.

If you are using Prometheus for monitoring and you're scraping the kube API servers, you can use the following PromQL query to monitor activity:

rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)

And if you want to compare the total rate of CSRs for 1.11 vs 1.12 kubelets in your cluster:

sum(rate(
  apiserver_request_count{
    client=~"kubelet.*", 
    resource="certificatesigningrequests",
    verb="POST"
  }[5m]
)) by (client)

[pichouk]

Hi,
I'm running a 1.12 EKS cluster with nodes on private subnet and I am facing the same issue. I guess this is because I have a Route53 private zone in the cluster.
Also, I can reproduce this behavior using AMS optimized AMI and custom (Debian based) AMI.

Do you have an ETA for a patch on this issue ?

[aviramradai]

Hi,

Whats the status of this issue?
Do you when you will release an update ?

[micahhausler]

Hi all, this has been resolved globally on the EKS side and no customer action is required. This is also fixed for instances in VPCs with custom DHCP option sets.

If you are using a custom non-EC2 assigned hostname, you'll need to configure the kubelet using the --hostname-override flag to an EC2-identifiable hostname (Ex: the instance's private DNS name) so that our certificate signer can verify that the kubelet certificate is for a legitimate target.

If you're still experiencing issues, please let us know.

[erks]

@micahhausler i think --hostname-override is currently ignored when --cloud-provider is specified.

[eviln1]

Hi @micahhausler ;
It seems that the eks.2 update didn't address this issue: kubernetes/kubernetes#77092
Do you have any insight on this ?
Thanks,

[micahhausler]

@eviln1 can you post the contents of one of your CSRs that is not getting approved?

kubectl get --raw /apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=1 \
    | jq -r .items[0].spec.request \
    | base64 --decode

[eviln1]

The request:

$ kubectl get csr vault-74975fcd67-ht6qg-vault-4iz5n -o jsonpath='{.spec.request}' | base64 --decode | openssl req -text -noout -in /dev/stdin
Certificate Request:
    Data:
        Version: 1 (0x0)
        Subject: CN = vault
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (3072 bit)
                Modulus:
                    00:cd:d0:9f:b7:2f:23:70:68:f4:8e:8f:03:75:e0:
                    cc:e2:11:68:36:a2:dd:d4:8f:91:97:05:9d:fd:ff:
                    08:9e:d5:79:df:a8:b1:f1:41:96:86:8a:43:43:55:
                    36:1d:cf:be:ec:b1:1a:cb:67:5e:04:16:8f:d4:25:
                    50:c8:92:8a:65:68:2f:02:d6:37:e3:b8:7e:05:d3:
                    18:ac:54:d3:c8:04:b3:9f:ea:70:db:e3:93:c8:de:
                    1c:93:bd:f9:d6:b8:5c:3c:1f:26:fd:f4:2d:af:25:
                    06:0b:9c:96:38:bc:e2:8e:a8:1d:43:58:4c:fa:e6:
                    75:c8:c4:95:2c:b4:78:85:7f:7b:d1:18:ce:6e:c2:
                    74:6a:8b:e1:7c:19:9b:76:f7:50:a9:6a:41:28:0c:
                    35:66:0d:f4:6d:69:e3:98:79:6a:5c:80:b3:3e:47:
                    cb:53:77:34:00:ee:a9:3e:22:41:b1:25:e1:6d:e5:
                    8c:ee:82:f8:b9:c0:98:5e:86:6f:9c:75:42:9a:e8:
                    2c:c6:56:f2:ef:14:5c:83:56:f9:ba:bd:a7:d8:ee:
                    38:f2:e2:57:79:66:e7:4a:51:ef:a9:ab:bc:4c:e0:
                    92:b1:91:c3:38:a0:ed:65:47:d4:64:4c:ec:42:20:
                    fc:d8:13:f7:5a:24:8c:b6:9d:0d:bf:93:ba:fb:ee:
                    bd:1b:be:12:9e:98:b8:e1:7f:07:06:fc:3a:e4:06:
                    43:5e:23:e7:ee:8b:60:c5:c0:38:85:e0:55:dd:f0:
                    a8:20:5d:3f:ed:7f:08:25:90:ff:da:99:ac:10:e3:
                    a5:70:bd:12:3b:c2:f7:de:51:3a:44:b3:7b:fd:c9:
                    94:be:78:97:a3:0e:53:86:38:a6:4f:8c:88:c0:e2:
                    58:54:20:22:1b:c8:07:6b:ce:f5:ba:46:a5:75:f6:
                    8e:bc:0e:fa:37:f4:4e:35:89:a5:1f:d0:67:12:bd:
                    2e:12:f0:c7:62:0b:e8:e8:76:67:83:2e:e6:49:c5:
                    40:84:83:71:c9:af:ca:f0:6c:9d
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name: 
                DNS:vault, DNS:vault.vault, DNS:vault.vault.svc.cluster.local, IP Address:172.24.18.208
    Signature Algorithm: sha256WithRSAEncryption
         bb:1d:e4:36:29:c9:86:b9:32:2a:47:6c:4d:16:89:1c:f5:1e:
         a2:e8:26:a5:cc:e7:f6:6b:44:ec:2c:cf:a7:88:2c:f3:21:15:
         14:d3:3c:d0:9c:16:9e:2d:50:7d:08:e9:db:79:44:de:b2:1a:
         02:65:2e:44:4a:be:9e:96:df:a9:f2:05:d5:8d:b7:65:a3:fb:
         74:cc:23:9f:ff:79:b0:8f:04:ad:c9:70:c5:af:fd:18:06:e8:
         d4:74:36:a1:d1:d7:6e:da:05:1d:c3:2f:38:34:55:b5:17:5e:
         51:e0:f5:09:6b:89:a5:8b:dd:f3:90:3d:64:e6:6a:ce:37:a2:
         c8:b5:ee:5f:2c:9a:9a:cb:aa:9b:52:be:e5:58:76:63:bf:bc:
         bd:bb:1a:ce:e6:f5:90:5d:02:66:9c:1d:c9:f0:2c:87:6e:ef:
         e4:8a:c5:21:55:54:b5:65:cb:e0:10:61:79:da:a0:74:4a:8b:
         bc:b9:4f:aa:31:c4:08:b9:ea:05:bc:4d:3f:d2:64:e3:c3:b7:
         f0:31:d8:b0:36:7a:d0:93:94:19:a9:a9:a6:43:57:9f:fd:41:
         ba:4f:48:8e:1c:24:48:47:1e:98:61:0b:bf:ed:2e:b6:a7:d3:
         02:0b:e6:5d:6c:c1:96:b8:4f:09:b7:e7:a6:08:7f:34:7f:38:
         0a:c4:58:00:b7:fa:03:fa:76:e7:34:fb:17:d2:c8:e1:02:5a:
         22:3c:fb:03:38:92:5b:6c:a4:80:c4:0d:7d:14:c1:91:4b:cd:
         b7:e1:9a:cc:3b:ae:b8:bc:25:9e:a8:95:1b:4e:e5:07:af:d8:
         8f:d8:0d:bc:34:84:16:e2:03:1d:99:eb:6b:60:1b:24:ee:d4:
         48:0a:5c:f7:6d:cf:7b:00:62:3b:6f:10:1b:a1:b5:5a:db:c7:
         ac:03:83:02:0e:1d:a0:14:ab:7f:99:bf:fb:c6:2d:a9:a3:18:
         96:9c:62:9d:a3:0d:5e:16:5a:58:c2:ed:42:d3:c1:5d:84:85:
         da:ff:c6:38:7e:78

The certificate:

$ kubectl get csr vault-74975fcd67-ht6qg-vault-4iz5n -o jsonpath='{.status.certificate}' | base64 --decode | openssl x509 -text -noout -in /dev/stdin 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7c:6a:14:3a:07:8c:bf:21:55:08:62:11:7e:2c:ca:aa:f0:ea:ee:fa
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May 22 16:56:00 2019 GMT
            Not After : May 21 16:56:00 2020 GMT
        Subject: CN = vault
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (3072 bit)
                Modulus:
                    00:cd:d0:9f:b7:2f:23:70:68:f4:8e:8f:03:75:e0:
                    cc:e2:11:68:36:a2:dd:d4:8f:91:97:05:9d:fd:ff:
                    08:9e:d5:79:df:a8:b1:f1:41:96:86:8a:43:43:55:
                    36:1d:cf:be:ec:b1:1a:cb:67:5e:04:16:8f:d4:25:
                    50:c8:92:8a:65:68:2f:02:d6:37:e3:b8:7e:05:d3:
                    18:ac:54:d3:c8:04:b3:9f:ea:70:db:e3:93:c8:de:
                    1c:93:bd:f9:d6:b8:5c:3c:1f:26:fd:f4:2d:af:25:
                    06:0b:9c:96:38:bc:e2:8e:a8:1d:43:58:4c:fa:e6:
                    75:c8:c4:95:2c:b4:78:85:7f:7b:d1:18:ce:6e:c2:
                    74:6a:8b:e1:7c:19:9b:76:f7:50:a9:6a:41:28:0c:
                    35:66:0d:f4:6d:69:e3:98:79:6a:5c:80:b3:3e:47:
                    cb:53:77:34:00:ee:a9:3e:22:41:b1:25:e1:6d:e5:
                    8c:ee:82:f8:b9:c0:98:5e:86:6f:9c:75:42:9a:e8:
                    2c:c6:56:f2:ef:14:5c:83:56:f9:ba:bd:a7:d8:ee:
                    38:f2:e2:57:79:66:e7:4a:51:ef:a9:ab:bc:4c:e0:
                    92:b1:91:c3:38:a0:ed:65:47:d4:64:4c:ec:42:20:
                    fc:d8:13:f7:5a:24:8c:b6:9d:0d:bf:93:ba:fb:ee:
                    bd:1b:be:12:9e:98:b8:e1:7f:07:06:fc:3a:e4:06:
                    43:5e:23:e7:ee:8b:60:c5:c0:38:85:e0:55:dd:f0:
                    a8:20:5d:3f:ed:7f:08:25:90:ff:da:99:ac:10:e3:
                    a5:70:bd:12:3b:c2:f7:de:51:3a:44:b3:7b:fd:c9:
                    94:be:78:97:a3:0e:53:86:38:a6:4f:8c:88:c0:e2:
                    58:54:20:22:1b:c8:07:6b:ce:f5:ba:46:a5:75:f6:
                    8e:bc:0e:fa:37:f4:4e:35:89:a5:1f:d0:67:12:bd:
                    2e:12:f0:c7:62:0b:e8:e8:76:67:83:2e:e6:49:c5:
                    40:84:83:71:c9:af:ca:f0:6c:9d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                35:30:06:9D:BE:82:12:AF:84:E2:06:88:C5:64:3F:29:FD:56:BE:D8
    Signature Algorithm: sha256WithRSAEncryption
         82:8b:ab:ce:e7:51:dc:6d:a1:11:4a:e8:7e:f8:13:73:24:aa:
         b1:73:ec:9b:c0:57:f6:06:29:3f:9c:c8:38:90:c3:8e:f2:15:
         d9:b3:81:ca:b8:db:59:09:ca:f9:83:8a:50:dc:d9:6b:32:87:
         33:a4:68:bf:cf:b9:1a:49:b3:ba:d2:f9:ab:9f:22:b9:ab:7f:
         76:b7:4b:18:ca:2e:51:51:69:3f:6b:69:90:9d:0c:31:9b:63:
         28:60:77:74:d4:b6:f0:02:b8:32:a1:fe:af:94:cf:8c:a4:86:
         ec:20:58:5a:53:9e:c3:47:f0:9d:fa:6b:01:6a:46:c5:93:2d:
         60:ef:ee:00:78:76:6a:5c:ef:4b:77:99:05:88:9a:8d:52:db:
         35:c3:ea:3f:ef:10:63:a0:37:f6:0d:e7:1c:cf:a5:12:38:bb:
         69:6d:9b:03:70:1b:0e:98:f5:8d:c5:d4:74:34:51:09:6c:c8:
         cd:68:3d:3f:ff:cd:e7:0a:5c:d5:16:e8:ba:f6:cd:5a:9c:34:
         35:bd:7c:f4:07:ae:49:c1:d3:f1:76:3e:c1:0f:73:9b:c4:80:
         57:1e:26:3a:4e:fe:24:29:57:12:e3:54:bb:bb:d9:d2:c2:49:
         66:44:28:2f:26:90:8f:a4:cc:4e:fb:88:f8:06:99:9a:7c:ff:
         cf:c9:af:b9

Additional info:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

[micahhausler]

Ok, so you're attempting to use the signer for a non-node server. Did this work previously for you in 1.11?

[eviln1]

Haven't tried it in 1.11 ;
As far as I understand, kubernetes/kubernetes#77092 wasn't about kubelet certificates; the example CSR requested the following SAN :

Requested Extensions:
  Subject Alternative Name:
    DNS:my-service, DNS:my-service.default, DNS:my-service.default.svc, DNS:my-service.default.svc.cluster, DNS:my-service.default.svc.cluster.local, DNS:localhost

[eviln1]

Hello,
do you have any updates @micahhausler ?
Thanks

[wph95]

hi @micahhausler met the same problem like eviln1 (not honor Subject Alternative Name in the originating CSR)
have any updates?