feat(translator): allow configuration of hostEnvKeys on WASM extensions

[sgargan]

exposes the hostEnvKeys configuration for WASM extensons through envoy extension policies. This enables access to env vars that are set on the host envoy processes and is a convenient way to share secret meterial with WASM extensions.

[daniel-white]

this may solve my #4424

[arkodg]

hostEnvKeys is pretty self explanatory, also brainstorming some more options

• vmConfig.envKeys
• env.Keys
• envKeys

[arkodg]

also linking some past work here from istio https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/#VmConfig

[zhaohuabing]

+1 to adding a vmConfig wrapper for better future-proofing, as additional VM settings may be needed in the future.
Should we also consider supporting explicitly specified key-value pairs?

vmConfig:
   env:
      hostEnvKeys:
        - host_key_1
        - host_key_2
      customEnvs:
        - key:  custom_key_1
          value: custom_value_1 
        - key:  custom_key_2
          value: custom_value_2    

[arkodg]

hey @zhaohuabing what value does customEnvs add that cannot be represented via config ?

[zhaohuabing]

Yeah, customEnvs just gives users more flexibility to pass configurations into the Wasm module. It can be handy for the Wasm modules just need some simple key-value pairs as input parameters. It's not a necessity though.

[sgargan]

hostEnvKeys is the name that envoy used for this and I figured it would make it easier for someone to join the dots from the envoy config if it matched here.

For a vmConfig wrapper, would this just contain the hostEnvKeys or the config block too? happy to add it, but is there value to the extra nesting?

[arkodg]

yeah I'm a +1 for hostEnvKeys, it leaves room for customEnvs . The other way to frame this is env.HostKeys, which leaves room for env.Custom.

[arkodg]

hey @sgargan, the PR looks great ! thanks for adding docs and test cases

added one comment around naming, if we get consensus from @envoyproxy/gateway-maintainers @envoyproxy/gateway-reviewers on naming, we could try and get this into the v1.2.0-rc.1 release (late next week)

[arkodg]

make gen-check is failing, you can run make generate && make manifests locally and commit those changes

[daniel-white]

it would be great to show an example here!

[sgargan]

👍 I've added an example to the envoyproxy customization docs and referenced it here.

[zhaohuabing]

@sgargan After discussing with @arkodg and @guydc in today's EG meeting, it was suggested to consider using Kubernets Secrets for the CustomEnvs to securely pass sensitive data to the Wasm modules. Additionally, ConfigMap should be supported as a storage option for non-sensitive configurations.

As the next step, I suggest raising a PR for the API design, followed by a follow-up PR for the implementation, once the community reaches a consensus on the API.

A similar example can be found in the CustomResponseBody API:

gateway/api/v1alpha1/shared_types.go

Lines 706 to 729 in 7188dad

	type CustomResponseBody struct {
	// Type is the type of method to use to read the body value.
	// Valid values are Inline and ValueRef, default is Inline.
	//
	// +kubebuilder:default=Inline
	// +kubebuilder:validation:Enum=Inline;ValueRef
	// +unionDiscriminator
	Type *ResponseValueType `json:"type"`
	// Inline contains the value as an inline string.
	//
	// +optional
	Inline *string `json:"inline,omitempty"`
	// ValueRef contains the contents of the body
	// specified as a local object reference.
	// Only a reference to ConfigMap is supported.
	//
	// The value of key `response.body` in the ConfigMap will be used as the response body.
	// If the key is not found, the first value in the ConfigMap will be used.
	//
	// +optional
	ValueRef *gwapiv1.LocalObjectReference `json:"valueRef,omitempty"`
	}

[arkodg]

@sgargan After discussing with @arkodg and @guydc in today's EG meeting, it was suggested to consider using Kubernets Secrets for the CustomEnvs to securely pass sensitive data to the Wasm modules. Additionally, ConfigMap should be supported as a storage option for non-sensitive configurations.

As the next step, I suggest raising a PR for the API design, followed by a follow-up PR for the implementation, once the community reaches a consensus on the API.

A similar example can be found in the CustomResponseBody API:

gateway/api/v1alpha1/shared_types.go

Lines 706 to 729 in 7188dad

	type CustomResponseBody struct {
	// Type is the type of method to use to read the body value.
	// Valid values are Inline and ValueRef, default is Inline.
	//
	// +kubebuilder:default=Inline
	// +kubebuilder:validation:Enum=Inline;ValueRef
	// +unionDiscriminator
	Type *ResponseValueType `json:"type"`
	// Inline contains the value as an inline string.
	//
	// +optional
	Inline *string `json:"inline,omitempty"`
	// ValueRef contains the contents of the body
	// specified as a local object reference.
	// Only a reference to ConfigMap is supported.
	//
	// The value of key `response.body` in the ConfigMap will be used as the response body.
	// If the key is not found, the first value in the ConfigMap will be used.
	//
	// +optional
	ValueRef *gwapiv1.LocalObjectReference `json:"valueRef,omitempty"`
	}

rethinking this one @guydc @zhaohuabing we can't allow the app dev who has access to the EEP to inject any env vars into the proxy, so we'll need to split up the task of injecting env vars (using EnvoyProxy resource by the platform admin) and allowing the app dev to use it by specifying this field

[sgargan]

rethinking this one @guydc @zhaohuabing we can't allow the app dev who has access to the EEP to inject any env vars into the proxy, so we'll need to split up the task of injecting env vars (using EnvoyProxy resource by the platform admin) and allowing the app dev to use it by specifying this field

there is definitely the danger that a dev with access to the EEP could exfil secrets that they shouldn't have access to if they could reason about what keys to use. This is mostly a problem if the EP is shared

Perhaps the EEP should only be able to use a secret from the same namespace that it explicitly calls out by name along with the key names to expose it to wasm? If the target Proxy is in a different namespace it might require duplicating the secret into that namespace with a reconciliation process to remove it when the referencing EEP is removed.

[sgargan]

As the next step, I suggest raising a PR for the API design, followed by a follow-up PR for the implementation, once the community reaches a consensus on the API.

Sure thing. Just so I'm clear on the process, are you suggesting a PR with just the wasm_types.go in it to capture the api conversation?

[arkodg]

hey @sgargan , @guydc and I discussed this issue again today in the community meeting and prefer if the API was env.HostKeys instead of hostEnvKeys .

It should be fine to make all the changes in this PR, we ideally prefer if the API and Implementation PR are separate to keep the PR sizes small and easier to review, but also the API discussion ( and churn) may cause a lot of work/change in the PR for the commiter

[daniel-white]

@arkodg i'm currently implementing a wasm plugin and i need to pass some sensitive values. is this likely going to land in 1.2?

[arkodg]

@arkodg i'm currently implementing a wasm plugin and i need to pass some sensitive values. is this likely going to land in 1.2?

this won't make it into v1.2, we just cut the RC release and can't add anymore API changes to this release, the workaround would be to use https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/

ChatGPT can be helpful here, path and name field will need to be rewritten/edited though

can you write a EnvoyPatchPolicy to set hostEnvKeys for wasm
ChatGPT said:

ChatGPT
To write an EnvoyPatchPolicy that sets the hostEnvKeys for a WebAssembly (Wasm) extension in Envoy Gateway, you would use the JSONPatch mechanism. The hostEnvKeys parameter is used to pass environment variables from the host to the Wasm runtime, enabling custom behavior based on external values.

Here's an example of how you can define the policy to add hostEnvKeys for a Wasm filter:

yaml
Copy code
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: wasm-env-keys-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: default
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/http_filters/0"
      value:
        name: "envoy.filters.http.wasm"
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm"
          config:
            vm_config:
              vm_id: "my_wasm_vm"
              runtime: "envoy.wasm.runtime.v8"
              code:
                local:
                  filename: "/etc/envoy/wasm/my-wasm-extension.wasm"
              environment_variables:
                key_values:
                  ENV_VAR_ONE: "value1"
                  ENV_VAR_TWO: "value2"
                host_env_keys:
                  - "HOST_ENV_VAR1"
                  - "HOST_ENV_VAR2"
Explanation:
The EnvoyPatchPolicy modifies the Envoy configuration using a JSONPatch to target the HTTP listener (default/eg/http) and insert a Wasm filter configuration.
The host_env_keys field within the vm_config block specifies which environment variables from the host should be passed to the Wasm VM. For example, HOST_ENV_VAR1 and HOST_ENV_VAR2 are included.
You can customize the paths and variable names as per your specific deployment.
Once this configuration is applied, your Wasm extension will have access to the specified host environment variables.

Make sure that the Wasm binary (my-wasm-extension.wasm) is correctly mounted in the expected location, and that the host environment variables (HOST_ENV_VAR1, HOST_ENV_VAR2) are set on the Envoy Gateway host.

This example gives you the flexibility to leverage host environment variables within your Wasm extensions, enabling custom logic based on external inputs

[zhaohuabing]

rethinking this one @guydc @zhaohuabing we can't allow the app dev who has access to the EEP to inject any env vars into the proxy, so we'll need to split up the task of injecting env vars (using EnvoyProxy resource by the platform admin) and allowing the app dev to use it by specifying this field

@arkodg Sorry, I didn't quite get the security concern behind this. Does this mean that we can still use a ValueRef to a local Secret, just like what EG does with the SecuritPolicy, but not an env from the host because the host envs are global scope and the app dev may not be allowed to access them?

[arkodg]

rethinking this one @guydc @zhaohuabing we can't allow the app dev who has access to the EEP to inject any env vars into the proxy, so we'll need to split up the task of injecting env vars (using EnvoyProxy resource by the platform admin) and allowing the app dev to use it by specifying this field

@arkodg Sorry, I didn't quite get the security concern behind this. Does this mean that we can still use a ValueRef to a local Secret, just like what EG does with the SecuritPolicy, but not an env from the host because the host envs are global scope and the app dev may not be allowed to access them?

yes, retrieving a value here requires inserting it to the host first (for hostEnvKeys, not customKeys), which means you could overwrite any existing val, that maybe critical for operation of the proxy or for another route

[daniel-white]

Thanks @arkodg - your sample got me started and i found the minimal patch required for those else interested. can't wait till this is a first class feature!

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: patch-name
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      name: ns/gc/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/http_filters/0/typed_config/config/vm_config/environment_variables"
        value:
          host_env_keys:
            - "ENV_VAR_1"
            - "ENV_VAR_2"

[zhaohuabing]

yes, retrieving a value here requires inserting it to the host first (for hostEnvKeys, not customKeys), which means you could overwrite any existing val, that maybe critical for operation of the proxy or for another route

Got it, since envs are used host wide, so it should be manged by the admin who has a global view to all variables. Thanks for the explaination!