Support for dynamic loading of Kong custom plugins

[sanjimoh]

Context

Based on my understanding of how currently any Kong plugins becomes usable is actually 2 step process -

1. Load Phase

• Upon a successful Load, a plugin is known to a Kong instance.
• It requires a restart of an already running Kong instance. This is because Kong currently doesn't support dynamic loading of plugin.

2. Enable Phase

• Upon enable, a plugin is now available for usage by Kong runtime constructs (routes/services).
• Pre-requisite for enable is a successful load.
• This is a runtime operation and doesn't require any restart.

The existing way, in Kubernetes world, for achieving above is by -

1. Creating a Configmap with custom plugin code in the Kong’s namespace
2. Load the above configmap on the Kong pod
3. then update the KONG_PLUGINS env to load this plugin.

Concern

OOB plugins or Kong bundled plugins are already loaded during the Kong bootstrapping because these are pre-bundled. However, the custom plugins are unknown to Kong instances; hence these requires to be loaded first which involves a restart of Kong instances.

This approach could be tricky in -

1. externally managed Kong APIGW scenario because the custom plugin provider may not have the necessary permission to provision anything in the Kong namespace.
2. Also, in the scenario where there is a single Kong instance running following the existing approach would lead to downtime.

Request

I was looking for a classical dynamic loading behaviour, wherein all that is required is to externalise the custom plugins onto some location (local/remote/url) already known to the running Kong instance. Kong looks for & lazily loads this plugin upon certain trigger (ex. during enabling phase of the custom plugin) without anyone requiring a need to provide the plugin name/details every time there is a new custom plugin. Hence, no restart of Kong instances would be required and would really help addressing the other concerns I had indicated earlier.

Curious to hear out what do you think about this?

[Tieske]

A restart is not required, a reload will do.

If the plugin has a custom DAO, then it would require a migration (DB updates) on the control plane. That's a complication factor. Also a DP needs to know at load time what plugins it has available, otherwise it cannot tell the CP which plugins it supports.

Now if the CP holds the plugins and pushes them to the DP, that is something that is on the radar already.

[sanjimoh]

@Tieske : I'm referring to KIC Kong Ingress Controller for Kubernetes setup.

A restart is not required, a reload will do.

[@sanjimoh] With the current design, how does one reload without a restart of the Kong pod(s)? The configmap consisting of the custom plugin code needs to be loaded and enabled which involves changing the Kong deployment manifest and thats going to restart the Kong pods.

Now if the CP holds the plugins and pushes them to the DP, that is something that is on the radar already.

[@sanjimoh] CP/DP segregation comes with Kong Hybrid deployment mode & from here, I learn that KIC supports hybrid deployment option as well.

So, when you said it's on the radar, is it something a work in progress item or its in the backlog and yet to be picked? Could you provide some reference to this item so that I can keep an eye on the same?

[oxie]

We face same issue when using Kong in Hybrid mode, we have our kong-CP i one aws eks cluster and kong-dp in another aws account and eks cluster. We cannot find an easy way to upgrade our plugins across the kong-DP without restaring the pods, which would serve live traffic and we have different K8S services, we were looking for a way to dynamically load the plugins on the kong-dp from the kong-cp but with no luck so far.

There is some minimal information on setting up a kong plugin server, but i cannot find any real documentation for it.
I am curious how usually people do this operation, it doesn't make sense we are the first or second people to hit this issue, it should be something already featured and working by my opinion, if this tool is supposed to be used in Prod envs.

Soo Any updates on this issue?

[kodeeswaran-malaimuthu]

@sanjimoh, Are you still facing this issue, we are starting the POC now.

[oxie]

we solved our issue by making a python script that uses a mounted shared efs across all our DP`s , we use jenkins pipeline to install the plugins to the efs, and the python is observing for file changes on the efs once it detects the changes, we added a config map lua block that when you hit localhost:2020/kong-reload, to execute lua os command kong reload, this is how we managed to sync dynamically all our custom plugins. We didnt saw better solution so far... and for our test so far this works flawlessly

[Tieske]

@sanjimoh

So, when you said it's on the radar, is it something a work in progress item or its in the backlog and yet to be picked? Could you provide some reference to this item so that I can keep an eye on the same?

Not in progress, but under consideration.

Also a more general note (on the approach by @oxie ); Adding plugins means adding executable code to a container. So rebuilding the image and redeploying is in line with best-practices. This means recreating a new container image, with the custom plugins embedded. No configmaps.

If you dynamically change the contents of the container, you're changing behaviour at runtime. This might be flagged in a security audit.

[oxie]

actually no, we are using the official kong helm chart, and yes we build only the custom image which is a python script observer of the shared efs , which is actually mounted as a sidecar container to the main deployment, and we created a ConfigMap with lua logic that whenever localhost:2020 is hit, it will execute kong reload command inside the kong-dp container, by using this low level system approach we are not redeploying our pods in this way and we guarantee that it will continue to work as expected and no discrepancies in traffic will happen everytime we add new plugins or remove older ones because it does graceful reload at system os level. when the observer python script detects changes on the kong shared plugins efs, i just issues command curl localhost:2020 to hit the main container.