Destination - Amazon Kinesis

[leggetter]

Publish events to Amazon Kinesis https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#API_PutRecord_Examples

TODO:

• QA
• Documentation

[alexluong]

I explored AWS Kinesis a bit and came up with this design document here. Can you take a look? The main 3 things to note here

• Kinesis doesn't support metadata natively
• How to decide partition_key
• Batch processing support (probably in a later iteration)

RFC

AWS Kinesis Destination Design

Overview

AWS Kinesis is a managed service for real-time data streaming that enables:

• Ingestion and processing of streaming data at scale
• Temporary storage of data (default 24h, configurable up to 365 days)
• Partitioned data flow through shards (capacity units)

Kinesis requires:

• Stream name
• AWS region
• Partition key strategy
• AWS credentials

Implementation Approach

Core Operation: PutRecord

For the initial implementation, the single-record PutRecord API will be used which:

• Sends one record at a time
• Requires stream name, partition key, and data payload
• Aligns with Outpost's current per-event publishing model

// AWS SDK PutRecord operation
PutRecordInput {
    StreamName: "my-stream",
    PartitionKey: "computed-key",
    Data: []byte(eventJSON)
}

Future optimization could include PutRecords for batching multiple events in a single call.

Metadata & Data Payload

Kinesis records are limited to 1MB and contain:

• Data payload (event data as JSON)
• Partition key (determines shard assignment)
• Sequence number (assigned by Kinesis)
• Timestamp (assigned by Kinesis)

Outpost event metadata will be included in the data payload itself.

{
  "metadata": {},
  "data": {}
}

Credentials

Required AWS credentials follow standard pattern:

{
  "key": "AWS_ACCESS_KEY_ID",
  "secret": "AWS_SECRET_ACCESS_KEY",
  "session": "AWS_SESSION_TOKEN" // Optional
}

• All fields should be treated as sensitive
• Session token is optional (used for temporary credentials)

Configuration

Essential configuration parameters:

{
  "stream_name": "my-kinesis-stream", // Required
  "region": "us-east-1", // Required
  "partition_key_template": "{{event.topic}}", // Optional, defaults to event.topic
  "endpoint": "https://kinesis.us-east-1.amazonaws.com" // Optional, for custom endpoints
}

Partition Key Importance

The partition key is critical for Kinesis as it:

1. Determines which shard processes the data
2. Controls distribution of events across shards
3. Maintains ordering within a partition

Partition Key Template

A templating approach would provide flexible partition key generation:

// Examples
"{{event.topic}}" // Group by event type
"{{event.data.user_id}}" // Group by user
"{{event.topic}}_{{event.data.region}}" // Composite key

Templating Options

Several template engines could be considered:

1. JMESPath (Recommended)

   • Industry standard adopted by AWS CLI and Azure
   • Purpose-built for working with JSON data
   • Cross-platform with implementations in multiple languages
   • Well-documented with formal specification
   • Example: join('_', [event.topic, event.data.user_id || 'default'])

2. JSONPath

   • Used extensively in Kubernetes
   • Good for data extraction from complex structures
   • Less expressive for string manipulation and combining values
   • May require additional string handling code
   • Example: $.event.topic + '_' + $.event.data.user_id

3. CEL (Common Expression Language)

   • Backed by Google, used in Kubernetes admission control
   • More powerful but heavier implementation
   • Full expression language rather than just templating
   • Provides type checking and security guarantees
   • May be overkill for simple templating needs
   • Example: event.topic + '_' + (has(event.data.user_id) ? event.data.user_id : 'default')

4. Handlebars

   • Limited Go implementations available
   • Better supported in JavaScript/web environments
   • Less ideal than the other options for backend Go services
   • Example: {{event.topic}}_{{#if event.data.user_id}}{{event.data.user_id}}{{else}}default{{/if}}

Future Considerations

PutRecord vs PutRecords

Initial implementation will use PutRecord for simplicity, but future optimization could include PutRecords for batching:

Feature	PutRecord	PutRecords
Records per call	1	Up to 500
Implementation	Simple	Complex (requires buffering)
Throughput	Lower	Higher
Latency	Higher per record	Lower per record
Error handling	Simple	Partial success possible

Additional Future Enhancements

• Configurable AWS SDK settings (timeouts, retry policy)
• Enhanced error reporting specific to Kinesis

[leggetter]

@alexluong - Thanks 👍

Can you update the RFC to provide an example of what the payload received by Kinesis would look like? Data + Metadata.

Since templating has been mentioned a few times, should we adopt JMESPath across the board? Is there any place that uses templating now that we should update? Since RabbitMQ has a routing key that seems to play a similar role to a partition key, should we change how the routing works there?

[alexluong]

Can you update the RFC to provide an example of what the payload received by Kinesis would look like? Data + Metadata.

Updated. And yes, data + metadata

{
  "metadata": {},
  "data": {}
}

---

Since templating has been mentioned a few times, should we adopt JMESPath across the board?

Potentially, although it's not as simple as that I think. There are 2 templating context here:

1: operator templating - folks who deploy Outpost

In this context, we're using Golang template because that's the language Outpost was built in. The DX is important but not as critical as people who operates Outpost is expected a certain level of technical capability.

Example: webhook signature header template

2: end-user templating - users of Outpost operator

In this context, we optimize for the best DX (with perf in mind of course) and therefore we don't expect people to pick up a more "obscure" language such as Golang templating.

Because of that, I tried to think of more standardized or user-friendly technologies like Handlebars or in this case, JMESPath.

Another reason JMESPath makes sense here is that AWS user is semi-expected to be familiar with JMESPath given that it's adopted in AWS CLI itself.

---

Since RabbitMQ has a routing key that seems to play a similar role to a partition key, should we change how the routing works there?

They're a bit different. RabbitMQ routing key is kind of like a topic in a PubSub system. I think giving the end-user the ability to customize the routing key would be cool, but I don't consider that a necessity. They can make it work with the Outpost way, for the most part.

On the other hand, the partition key is critical in AWS Kinesis or data streaming technology. It may be different from case to case, depending on the traffic pattern and workload, and therefore must be as flexible as possible. I'm happy to elaborate further on this later in our call if that helps.

[laurenzlong]

This is a great proposal! I like the idea of a templating approach to partition selection. Could the event metadata also be used in the template?

[leggetter]

@laurenzlong

This is a great proposal! I like the idea of a templating approach to partition selection. Could the event metadata also be used in the template?

Yeah, the event structure is as follows so you should be able to do event.metadata within the template:

{
  "tenant_id": "<TENANT_ID>",
  "destination_id": "<DESTINATION_ID>", // Optional. Provide a way of routing events to a specific destination
  "topic": "topic.name", // Topic defined in TOPICS environment variable
  "eligible_for_retry": false | true, // Should event delivery be retried. Default is false.
  "metadata": Payload, // can by any JSON payload,
  "data": Payload // can by any JSON payload
}

Do you have any thoughts or feedback on the payload structure that reaches Kinesis? Current the suggestion is as follows because it has no in-built concept of metadata unlike RabbitMQ (headers) or SQS (message parameters).

{
  "metadata": {},
  "data": {}
}

[laurenzlong]

Hey @leggetter I think the proposed payload makes sense. Even more ideal would be if there's a configuration option for whether to have metadata in the payload (some customers may not have any metadata, so it's inconvenient to always have to read into the nested field for data for get the data.):

{
  "stream_name": "my-kinesis-stream", // Required
  "region": "us-east-1", // Required
  "partition_key_template": "{{event.topic}}", // Optional, defaults to event.topic
  "endpoint": "https://kinesis.us-east-1.amazonaws.com" // Optional, for custom endpoints
  "metadataInPayload": true // Optional, defaults to true
}

Then if it is false, the payload would only include what's data. But this is a nice to have feature. I think always having the metadata in the payload would also work.