feat: Support AWS Kinesis destination

[alexluong]

This implements the RFC from #305. A few note-worthy things to call out here:

1. Add config.metadata_in_payload bool config as suggested in the issue.

2. config.endpoint UI

This should probably be hidden. However, it would make local dev testing difficult, especially when using the UI to test locally. With SQS, we can derive the endpoint from the queue_url field and detect whether it's production AWS or local LocalStack. We can't do the same with AWS Kinesis.

Options:

1. Hide it and assume production AWS
2. Show it and provide default to AWS

3. Partition key template payload

This PR implements the partition templating with JMESPath as discussed in the RFC.

One thing to note is that I ended up going with the message payload as the input for the template. What I mean is that when producing a Kinesis record, we need 3 info: stream_name, partition_key, and data (bytes version of a Go map); whatever the data content is, that map is the payload for the partition key template.

For example, with an event like so:

{
    "id": "evt_1",
    "tenant_id": "te_1",
    "topic": "user.created",
    "eligible_for_retry": false,
    "metadata": {
        "mymeta": "metadata"
    },
    "data": {
        "data": "data"
    }
}

If metadataInPayload: true, the resulting payload would be:

{
  "metadata": {
    "mymeta": "metadata",
    "event-id": "evt_1",
    "topic": "user.created",
    "timestamp": "123456"
  },
  "data": {
    "mydata": "data"
  }
}

So, you can do something like metadata.mymeta or metadata.topic as the partition key tempalte.

If metadataInPayload: false, the resulting payload would be

{
  "mydata": "data"
}

In that case, you can do mydata for the template. Or an empty string, in which case we'll use the event ID as the partition key. One note is that you will NOT be able to use topic as the partition key in this case, unless topic is part of the data itself.

Reasoning

The reason I went with this approach was around the end-user perspective. They won't know what an event is, and the only thing they know is the payload they'll receive. Therefore, I think this makes a bit more sense than the incoming event payload from the producer/operator side.

However, this still requires the user to know beforehand what the expected data shape is. It's a bit more complicated than webhook, for example, where you can just provide a URL and observe the incoming requests. Maybe it's a necessary consideration, given the technology.

4. Delivery details

This is how the UI looks right now.

NOTE: The "message" UI is incorrect now. Currently, the message UI is fake, deriving from the event. For Kinesis, it doesn't account for the fact that the user can opt-out of metadata, and therefore the message can be different. We probably need to add a task to persist message data & display properly.

---

Dev notes:

• When testing locally, I added a script to help provision a stream and log incoming events: go run ./cmd/destinations/awskinesis. Make sure you run make up/deps beforehand. There're new changes to the Docker Compose, so if you had it running previously, also make sure to restart the containers.

---

TODO:

• QA & go over DX/UX
• Instructions & re-write config descriptions / helper texts

---

Next step:

• Consider PutRecords to send events in batch
• Consider ways to improve UX around templating

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
outpost-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 23, 2025 2:41am
outpost-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 23, 2025 2:41am

[leggetter]

1. config.metadata_in_payload

I don't think this should be exposed to the end-user. I think it's operator-only config as the operator should determine what's in the event payload and not the end-user.

2. config.endpoint UI

Show it for now.

3. Partition key template payload

I don't quite understand this. But I'm assuming my comment on 1. would have an impact.

May be something worth discussing live and then capturing the outcome in GitHub.

4. Delivery details

Gotcha. End logic may be impacted by 1. and 3.