Skip to content

PXY-1722 Dynamic headers based on the payload fields #727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: console-1.34_gateway-3.9.0
Choose a base branch
from
Open

PXY-1722 Dynamic headers based on the payload fields #727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 74 additions & 5 deletions docs/gateway/interceptors/data-security/dynamic-header-injection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,58 @@ Here are the values we can expand
- apiKeyVersion
- timestampMillis

### Record Content Variables (v3.9+)

- `{{record.key}}` - Extract the entire key payload as a string
- `{{record.value}}` - Extract the entire value payload as a string
- `{{record.key.fieldName}}` - Extract a specific field from the record key
- `{{record.value.fieldName}}` - Extract a specific field from the record value

## Configuration

| config | type | description |
|:-----------------|:--------|:------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Regular expression that matches topics from your produce request |
| headers | Map | Map of header key and header value will be injected, with the header value we can use `{{userIp}}` for the user ip information we want to be injected |
| overrideIfExists | boolean | Default `false`, configuration to override header on already exist |
| config | type | description |
|:---------------------|:-------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Regular expression that matches topics from your produce request |
| headers | Map | Map of header key and header value will be injected, with the header value we can use `{{userIp}}` for the user ip information we want to be injected |
| overrideIfExists | boolean | Default `false`, configuration to override header on already exist |
| schemaRegistryConfig | [SchemaRegistry](#schema-registry-configuration) | Configuration of your Schema Registry, required if you want to inject headers into data produced using Avro, JSON or Protobuf schemas |

## Schema Registry configuration

As soon as your records are produced using a schema, you have to configure these properties in your encryption/decryption interceptors after `schemaRegistryConfig` in order to (de)serialize them. Gateway supports Confluent-like and AWS Glue schema registries.

| Key | Type | Default | Description |
|-----------------------|--------|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](#use-environment-variables-as-secrets) |
| **Confluent Like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1` |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`) |
| `basicCredentials` | string | | Access credentials for AWS (see below section for structure) |
| **AWS Credentials** | | | **AWS Credentials Configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | `true` / `false` flag to determine whether the credentials provided should be validated when set. |
| `accountId` | string | | The Id for the AWS account to use. |


If you do not supply a `basicCredentials` section for the AWS Glue schema registry, the client we use to connect will instead attempt to find the connection information is needs from the environment, and the credentials required can be passed this way to the Gateway as part of its core configuration. More information on the setup for this is found in the [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

## Use environment variables as secrets

You probably don't want your secrets to appear in your interceptors. In order to make sure this doesn't happen, you can refer to the environment variables you have set in your Gateway container.

For that, you can simply use the format `${MY_ENV_VAR}`.

We recommend you use this for Schema Registry or Vault secrets, and any other values you'd like to hide in the configuration.

## Example

### Simple header injection

```json
{
"name": "myDynamicHeaderInjectionInterceptor",
Expand All @@ -52,6 +94,33 @@ Here are the values we can expand
}
```

### Header injection based on the payload fields

```json
{
"name": "myDynamicHeaderInjectionInterceptor",
"pluginClass": "io.conduktor.gateway.interceptor.DynamicHeaderInjectionPlugin",
"priority": 100,
"config": {
"topic": "topic.*",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"additionalConfigs": {
"schema.registry.url": "${SR_URL}",
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
},
"headers": {
"X-CLIENT_IP": "{{userIp}} testing",
"X-USER-ID": "{{record.key.id}}",
"X-USER-EMAIL": "{{record.value.email}}"
},
"overrideIfExists": true
}
}
```

Let's produce a simple record to the `injectHeaderTopic` topic.

```bash
Expand Down
30 changes: 30 additions & 0 deletions src/pages/changelog/Gateway-3.9.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,37 @@ tags: features,fixes

- [Breaking changes](#breaking-changes)
- [New features](#new-features)
- [Dynamic Header Injection from Record Values](#dynamic-header-injection-from-record-values)


### Breaking changes

### New features

#### Dynamic Header Injection from Record Payloads

The Header Injection Interceptor has been enhanced to support deriving header values directly from record payloads.
This powerful feature allows you to extract:
- The entire record key or value and inject it as a header
- Specific fields from record keys or values inject them as headers

You can now reference record fields using mustache syntax:
```json
{
"config": {
"topic": "topic.*",
"headers": {
"X-CLIENT_IP": "{{userIp}} testing",
"X-USER-ID": "{{record.key.id}}",
"X-USER-EMAIL": "{{record.value.email}}"
},
"overrideIfExists": true
}
}
```

This feature supports:
- Extracting values from JSON, AVRO, PROTOBUF serialized records
- Accessing record fields using dot notation
- Referencing the entire key or value payload
- Using mustache syntax for dynamic header values