Install "Custom CA Trust" on AKS nodes

[jabbera]

Private clusters cause AKS internet traffic to be directed out customers internet edge devices. We have a decrypting firewall. Therefore we need our Certificate Authority root cert installed on every node in our cluster or every package mirrored to our local repo as to not be intercepted by our MITM firewall. I was originally going the mirroring route until coredns broke last night during what I assume to be an upgrade, took down the whole cluster, and I didn't know how to update the image location. (Is this possible in AKS?)

Edit: Actually this is WAY worse for a new cluster. The deployment failed and I had to go in and dig around to find this:

Now I have a bootstrapping issue. I can't deploy a cluster without my CA installed but my automation that installs my CA doesn't run till after the cluster is provisioned. Luckily I was able to apply the CA installation daemonset described below, re-run my deployment and my cluster ended up in an alright state.

Customizing the CA repo should be built in node customization functionality to private clusters instead I had to find and follow these directions as well as augment them to be sure they run on every node: http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html

Please make this core functionality!

Hi jabbera, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

Triage required from @Azure/aks-pm

[miwithro]

Thanks for the request. Putting this on our backlog.

[alxzndr]

We face the same issue since to move to containerd.

On previous versions with docker we were able to configure our registry ca trough a daemonset.
This daemonset puts ca config files for our container registry in the /etc/docker/certs.d folder.
Docker automatically picked up these cert files.

Containerd has a similar functionality but this requires the registry configpath parameter to be set.
By default this parameter is empty on the AKS worker nodes.

If this registry config_path parameter is set to a default value we would not need daemonsets with elevated rights to be able to restart containerd.

For example:

  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

[jabbera]

Just to add this SIGNIFICANTLY increases node startup time. The daemon set has to run then all the core images need to be pulled. I'm averaging 8 minutes for node startup. It's sad:-(

[AndreasMWalter]

Hey @palma21 we have a customer that uses TLS inspection on their Checkpoint Cloudguard and we encounter issues when building AKS with TLS inspection enabled. Will this feature also enable nodes to customize the OS's Trusted Root Certificates?

Since provisioning Ephemeral Nodes seems to fail even before images for the system services are being pulled and the nodes never reach the ready state.

[AndreasMWalter]

@jabbera
We were just evaluating another workaround, which at preset seems to work however it is really... uhm... botched:
With the AKS Proxy Feature enabled we have created a dummy proxy entry and noProxy "*" and injected an onpremises Trusted Root CA. That seems to work, however obviously the AKS Proxy Feature is still in preview and this workaround is also probably not supported by Microsoft. (And might cause other issues)

{
    "httpProxy": "http://dummy.proxy/",
    "httpsProxy": "https://dummy.proxy/",
    "noProxy": ["*"],
    "trustedCa": "BASE64ENCODEDROOTCA"
}

[jabbera]

@AndreasMWalter this is genius. I'd love an official statement from MSFT about it.