Support single controller deployments

[hbagdi]

Problem Statement

Currently the ingress controller is only capable of deploying configuration to one Kong Gateway at a time, the consequence of which is that to horizontally scale the data-plane in DBLESS mode you must create an ingress controller for each gateway which has some performance costs and wastes associated with it. The purpose of this issue is to make it possible to configure the KIC with multiple Gateways so that they all receive the same configuration updates.

Proposed Solution

• Add functionality in the dataplane.KongClient (or similar struct with a superset of KongClient's functionality) to provide multiple Admin API endpoints and update each in c.Update() #3209
  • Above introduced check that the underlying Gateways use the same db setting but did not account for the fact that this issue wants to introduce this functionality only for dbless nodes. Single controller deployments: limit functionality to dbless only #3401 tracks that
• Single controller deployments: Admin API endpoints tracking via EndpointSlice controller #3400
• Ensure that tests for single controller deployments exist #3429
• When working with horizontally scaled KIC with more than 1 replica, leader election works and only 1 KIC sends configuration payloads to Kong Gateways #3528

Additional information

• Previous iterations of this issue described adding mTLS auth between the KIC and Gateways, but this was already resolved separately as part of mTLS Kong Admin API authentication support #1958
• Horizontal scaling options where each Gateway has the same configuration is covered by this scope
• Multi-tenancy options where each Gateway has it's own separate configuration are not covered by this scope if you're interested in such a feature please file a separate issue

Acceptance Criteria

• as an end-user of the KIC when using DBLESS mode Kong Gateways as my data-plane I should be able to scale the data-plane horizontally with a single instance of the control-plane.
• Gateway discovery in the Helm chart does TLS encryption by default #3618
• the KEP has been updated and moves to "implemented" state: https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md
• whatever methods of managing configuration for multiple Kong Gateways is implemented as part of this issue it also adds
  • Gateway discovery documentation #3619
  • Single controller deployments support in kong helm chart charts#710
  • Make all-in-one-dbless use SCD #3423

[seh]

We are definitely interested in this capability. If you need help, please let me know.

[marcoam]

+1

[shaneutt]

Plan:

1. proxy containers will now have their own pod in the Kong Ingress Controller
2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
3. service discovery
4. reconciliation needs to operate on N proxies now
5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

• mTLS is on Kong side
• need to update kong/kong-operator, kong/charts

[shaneutt]

After discussing further with @mflendrich we're going to take some time to do some further investigation on the problem domain here (as opposed to going right into implementation as currently written) as we feel that there's some potential for large gains in rethinking our approach to this crossroads in how we deploy and manage the proxy.

[shaneutt]

The team has decided that we'll hold on this issue for now as we try to make some architectural changes to the KIC which will ultimately make implementing this easier.

[swapnilpotnis]

Plan:

1. proxy containers will now have their own pod in the Kong Ingress Controller
2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
3. service discovery
4. reconciliation needs to operate on N proxies now
5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

• mTLS is on Kong side
• need to update kong/kong-operator, kong/charts

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service??
If so, then could you please let us know if https://docs.konghq.com/kubernetes-ingress-controller/1.1.x/guides/upstream-mtls/ is something different than what is being implemented??

We basically have a use-case where we have an outside service communicating with kong running over AKS. Now we wish to establish mTLS between the two. As we have installed kong from helm charts, we were not able to accomodate the variables as mentioned in the above given link. Hence wanted to know if it was possible with the current kong release or is the mTLS support by kong yet to be implemented?

[shaneutt]

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service??

In this particular case:

This covers mTLS between the Kubernete Ingress Controller Pod and access to the Admin API in the Kong Proxy Pod. Note that historically the containers have been in a single pod together and used localhost connectivity so mTLS was not enabled.

This does not pertain to any other services (e.g. the services being proxied to) for this scope.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaneutt]

Several weeks ago we started some work on this but it was not finished due to a desire by the maintainers to wait until KIC 2.0 which will bring a major re-architecture that would better support this feature.

We'll consider this blocked on the KIC 2.0 milestone for now and revisit once that is completed.

[pmalek]

This is currently being worked on.

#3268 aims to enable rendering and sending configs for multiple Kong Gateways: this will be done via the same flag that's in place today --kong-admin-url (by passing multiple, comma separated values).

When that's done then we'll work on making this not be bound to controller's Pod's lifetime, specifically to use a label selector for Kong's Admin API Service.

If all that gets released we'll be able to then incorporate those changes into helm chart so that users can utilize that deployment method to deploy single controller with multiple Gateways.

[mflendrich]

The remaining work here is to complete all the remaining checkboxes in the AC.

[czeslavo]

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

[pmalek]

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

For posterity: as agreed on sync meeting #3603 will track this effort but it won't be land in v2.9.

[mflendrich]

The core has been implemented and will ship in KIC 2.9 and the Helm chart 2.17.

There are a few open items left:

• Single controller deployments: limit functionality to dbless only #3401
• Gateway discovery in the Helm chart does TLS encryption by default #3618
• Make all-in-one-dbless use SCD #3423
• Gateway discovery documentation #3619
• Add envtest based tests for KongAdminAPIServiceReconciler #3568
• When gateway discovery is enabled, the liveness probe should be 👍 even if the adminapi service has 0 ready endpoints #3592
• When Gateway discovery is enabled and Kong is DB backed, KIC only sends config to one of Kong gateway endpoint #3606
• Gateway discovery is documented on docs.konghq.com #3604

Now that we have only those details to polish, I'm closing this huge issue in favor of the more targeted ones above.