Add support for preinstalled/trusted exec KRM plugins for GitOps tools

[mprahl]

The exec plugin style allowed for the plugin to be placed in a directory outside of the Kustomize path. This allowed for GitOps based workflows that support Kustomize (e.g. ArgoCD, Red Hat Advanced Cluster Management) to have Kustomize plugins preinstalled that are trusted and supported by the administrator or GitOps tool. For Red Hat Advanced Cluster Management, an exec plugin comes preinstalled and is usable in the GitOps workflow as documented here. We would like to continue a similar approach with the Exec KRM functions.

Based on my understanding of the documentation of the exec KRM plugins, it seems that the plugin must be in the same directory tree as the Kustomize configuration. This means that the GitOps tool must perform some kind of hack to copy the preinstalled plugin into the Kustomize configuration directory and hope that the user specified the correct path in their config.kubernetes.io/function annotation.

This also means that if a GitOps tool enables Kustomize plugins, it is susceptible to arbitrary code execution since a user can now just include any binary in their Kustomize configuration repo to be run as a plugin. Requiring that exec plugins be in a directory tree outside of the user provided Kustomize configuration guarded against this.

I see two options to rectify this:

1. Add a configuration option/environment variable that sets the directory where plugins are located and the user specifies the plugin by name rather than by path in their config.kubernetes.io/function annotation.
2. Add an allow list option of plugins that can be executed either by name or by digest. This would at least allow the hacky approach of copying the preinstalled plugin into the user provided Kustomize configuration repo when cloned down without exposing the GitOps tool to running a binary provided by the user.

[k8s-ci-robot]

@mprahl: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[natasha41575]

I believe if you just put the name of the exec file in your annotation, and put the exec file in a folder in your $PATH variable, kustomize can find it. @KnVerey Are we still planning to support that moving forward?

[KnVerey]

Thank you for the detailed description of your use case! This is definitely something we want to support in the KRM Functions version of exec extensions.

Re: arbitrary code execution, in the current release this is controlled by the fact that exec functions must be enabled explicitly in the Kustomize invocation, which enables the actor controlling that invocation to disable them globally. I completely agree that this is inadequate for your use case, where the end user and Kustomize executor are separate actors, so you definitely want to enable specific plugins only.

The current proposal to address this uses the upcoming KRM Functions Catalog feature, and it works very similarly to your option 2: the actor in control of the kustomize invocation provides a list of plugins they want to allow as part of that invocation. In the case of exec, Kustomize will retrieve and verify the binary before executing it--it is neither pre-installed (though we could add caching) nor controlled by the end user. Notably, the end user will no longer be able to point their configuration to an executable other than redundantly; lookup is entirely based on finding the GVK of the resource (i.e. policy.open-cluster-management.io/v1 PolicyGenerator in your case) in the administrator-controlled catalog.

These plans are still provisional, and we would greatly appreciate your input on them.

I believe if you just put the name of the exec file in your annotation, and put the exec file in a folder in your $PATH variable, kustomize can find it. @KnVerey Are we still planning to support that moving forward?

Related to my explanation above, I would prefer to be as explicit as possible and only support specific local paths within the Kustomization directory (to support ad hoc plugins for smaller scale use cases) and plugins retrieved from a published registry. Having a catalog entry be able to allowlist any binary by a certain name feels kinda weird to me, even if we require a sha to be provided. It would enable allowlisting well known binaries that are not actually plugin providers at all, which would be a terrible idea for security (though it would have to be done explicitly, at least). But I am open to changing my mind on this.

Provisional KEPS for reference:

• Catalog: Kustomize KRM Plugin Catalog kubernetes/enhancements#2906
• Plugin graduation: Kustomize Plugin Graduation kubernetes/enhancements#2953

cc @jeremyrickard

[mprahl]

Thanks for the detailed reply and helping me out with my use-case! I'll reply in-line.

Thank you for the detailed description of your use case! This is definitely something we want to support in the KRM Functions version of exec extensions.

Re: arbitrary code execution, in the current release this is controlled by the fact that exec functions must be enabled explicitly in the Kustomize invocation, which enables the actor controlling that invocation to disable them globally. I completely agree that this is inadequate for your use case, where the end user and Kustomize executor are separate actors, so you definitely want to enable specific plugins only.

The current proposal to address this uses the upcoming KRM Functions Catalog feature, and it works very similarly to your option 2: the actor in control of the kustomize invocation provides a list of plugins they want to allow as part of that invocation. In the case of exec, Kustomize will retrieve and verify the binary before executing it--it is neither pre-installed (though we could add caching) nor controlled by the end user. Notably, the end user will no longer be able to point their configuration to an executable other than redundantly; lookup is entirely based on finding the GVK of the resource (i.e. policy.open-cluster-management.io/v1 PolicyGenerator in your case) in the administrator-controlled catalog.

These plans are still provisional, and we would greatly appreciate your input on them.

I think this could work. I skimmed through kubernetes/enhancements#2908 and since the PR is already merged, I'll ask my question here.

Will a custom catalog for the exec functions support a local filesystem for the plugin path rather than HTTP(S)? For disconnected environments (i.e. no internet access), this would be beneficial to reduce the need of hosting an internal web server merely for this purpose.

In my situation, Kustomize is executed inside of a container running on Red Hat OpenShift or Kubernetes that may not have internet access and thus would have to provide a web server with the catalog on the cluster. It also would not have the privileges to run a container inside of a container, so the "container provider" would not work here, even if the image was loaded to the internal registry.

I believe if you just put the name of the exec file in your annotation, and put the exec file in a folder in your $PATH variable, kustomize can find it. @KnVerey Are we still planning to support that moving forward?

Related to my explanation above, I would prefer to be as explicit as possible and only support specific local paths within the Kustomization directory (to support ad hoc plugins for smaller scale use cases) and plugins retrieved from a published registry. Having a catalog entry be able to allowlist any binary by a certain name feels kinda weird to me, even if we require a sha to be provided. It would enable allowlisting well known binaries that are not actually plugin providers at all, which would be a terrible idea for security (though it would have to be done explicitly, at least). But I am open to changing my mind on this.

I agree that the allow-listing of a name (I meant a relative path but didn't explain it well) is odd outside of my explicit use-case where I would have complete control of the user provided Kustomization directory before executing Kustomize.

In regards to your comment of "It would enable allowlisting well known binaries that are not actually plugin providers at all, which would be a terrible idea for security". I don't really see how this is any different than just including the well known binary (or symlink) inside the Kustomization directory. I think allowlisting an absolute path could help alleviate this issue and the user's annotation could just specify the binary name which Kustomize could then compare against the configured allowlist.

Provisional KEPS for reference:

• Catalog: Kustomize KRM Plugin Catalog kubernetes/enhancements#2906
• Plugin graduation: Kustomize Plugin Graduation kubernetes/enhancements#2953

cc @jeremyrickard

[KnVerey]

Will a custom catalog for the exec functions support a local filesystem for the plugin path rather than HTTP(S)? For disconnected environments (i.e. no internet access), this would be beneficial to reduce the need of hosting an internal web server merely for this purpose.

I'll admit we weren't thinking about disconnected environments in the plan so far. Local filesystem is supported in the proposal, but subject to the load restrictor. That means your only option would be to symlink the binary into the Kustomization directory to make it "belong" to that Kustomization (and be able to be referenced by a relative path the load restrictor will accept).

In my situation, Kustomize is executed inside of a container running on Red Hat OpenShift or Kubernetes that may not have internet access and thus would have to provide a web server with the catalog on the cluster. It also would not have the privileges to run a container inside of a container, so the "container provider" would not work here, even if the image was loaded to the internal registry.

Good context, thank you.

I think allowlisting an absolute path could help alleviate this issue and the user's annotation could just specify the binary name which Kustomize could then compare against the configured allowlist.

I'll need to think about this a bit more, but allowing both absolute paths that are not subject to the load restrictor and relative paths that are could be a reasonable option. I'll add this to my list of things to address in the next iteration of the KEP (probably in a couple of weeks tbh).

Note that the user won't need an annotation at all anymore once that KEP is implemented: they will just provide the resource, and Kustomize will look up the implementation (binary in this case) in the catalog using the resource's GVK alone. This decoupling of the extension implementation from the end-user config is one of the benefits of Catalog. :)

[mprahl]

Thanks for your help @KnVerey! I look forward to hearing back on the next iteration. 🙂

[KnVerey]

Please see the proposal here: https://github.com/kubernetes/enhancements/pull/3097/files#diff-da31478d2b3a925c17471e989c953539b508eed0aae19d624ad943d08f4dd910R288

[snuggie12]

As I was reading this I mistook it for an idea I had about all of this so wanted to run it by you.

I was bummed to hear that helm is likely getting removed and being turned into a KRM function. Like a lot of other people who use argocd or even on the laptop the docker daemon is not present and in some cases will never be. This will leave most of us with old or forked versions of kustomize.

Would there be any possibility of actually pre-installing golang plugins into kustomize so that when compiled certain plugins are treated as built-ins? Essentially 3 tiers of plugins:

• core which can still be tightly coupled with kubectl
• semi-official ones like helm which could still be compiled in your repo as "kustomize-full" or something
• a 3rd option for people to put their own custom ones in a directory and can compile themselves

[KnVerey]

Kustomize does currently support Go plugins, but we are deprecating them as part of KEP 2953 for reasons explained in the KEP. After that, "built in plugins" will move to a more straightfoward setup, and it will no longer be possible to compile in any external plugins. However, it will be possible to compile a Catalog into Kustomize such that it pre-trusts certain plugins and will download and execute them transparently as needed.

The Helm transformer is moving to the new KRM Functions registry (WIP repo / KEP), which is planning support for exec functions in its beta phase. Once that is implemented, it will be possible to publish functions from the registry as executables that end users can trust via a Catalog. This hasn't been discussed for Helm specifically AFAIK (cc @natasha41575 ).