KIC: investigate if Kong can do mTLS on the Admin API or not

[hbagdi]

Currently, there is a controller is deployed alongside each proxy container of a kong inside the same pod. This works well for some cases but it will be more efficient and operationally sustainable if controller can be deployed as a pod and the proxy is deployed as a daemon set which can be scaled as needed. This is currently possible for kong deployments that are powered by a database but not in the DB-less mode.

To get this to work, Admin API needs to be exposed outside the pod, and it should be authenticated as well.

The task here is to investigate if Kong's Admin API can support mTLS using Nginx directive injection or not. If yes, test it. If all looks good, then we will create new issues to implement mTLS-based authentication between the controller and Kong's proxy pods.

[rainest]

This is simple enough to do with directive injection, which we can use to take advantage of NGINX's standard client certificate verification:
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_cert
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client

In kong.conf:

nginx_admin_ssl_client_certificate = /etc/kong/admin-client-ca.pem
nginx_admin_ssl_verify_client = on

This gist contains the test certs used and instructions for creating them using cfssl. Creating and signing new certs isn't really necessary (the existing test certs look fine), but it may be useful to show how:
https://gist.github.com/rainest/ac98c24932f7acb8ebf27ba4d1887b3c#file-signing-instructions-txt

With the certs in place, this shows the test procedure:
https://gist.github.com/rainest/ac98c24932f7acb8ebf27ba4d1887b3c#file-test-txt

Challenges with this approach:

• It's NGINX-native, not Kong-native, though the injected directives make it close enough. No Admin API creation of CA certificates or usage of them, but that's probably desirable for certs that protect the Admin API. Some integration on our end would be great to provide an alternative to RBAC tokens, but that's a distant dream.
• Because it's NGINX-native, the CA file must be present on the filesystem. We cannot load it via an environment variable, which is less convenient for Kubernetes. We can take advantage of Secret loading in the Helm chart, though note that this should provide the CA cert to Kong only, not its key.

[hbagdi]

Thank you for working on this, @rainest and this is great news.

It's NGINX-native, not Kong-native, though the injected directives make it close enough. No Admin API creation of CA certificates or usage of them, but that's probably desirable for certs that protect the Admin API.

I don't think either of these are limitations. We can achieve our goal without these. (I know you were being explicit and noting the details, I'm doing the same).

Because it's NGINX-native, the CA file must be present on the filesystem. We cannot load it via an environment variable

This does make it a little heavier than it should be but once set up, these pain should go away. We should be able to solve this problem with good documentation.
Kong should also support this out of the box but that's not part of the roadmap currently.

The scope of the ticket to investigate, which is complete and I'm going to close this issue.
#702 is now unblocked.