awskinesisexporter: Add support for partitioning records by traceId

[preetsarb]

Component(s)

exporter/awskinesis

Is your feature request related to a problem? Please describe.

Currently spans of a single trace can be spread over multiple shards, so its not possible (or require peer forwarding of spans) to perform tail sampling when consuming data from kinesis.

Describe the solution you'd like

Kinesis supports partitioning of records. traceId can be used as the partitionKey so all spans are routed to the same shard.

Describe alternatives you've considered

No response

Additional context

Similar feature was added for kafka exporter: #12318

[github-actions]

Pinging code owners:

• exporter/awskinesis: @Aneurysm9 @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jamesmoessis]

@preetsarb have you considered using layered collectors connected with load balancing exporter to shard by trace ID? Could be an option in your interim.

Another problem I foresee with this is the Kinesis limit of 1MB per second per shard. Traces can easily be over 1MB so it begs the question how can that limit be avoided at scale.

[preetsarb]

@jamesmoessis we have been using something similar to load balancing exporter in our trace processing stack, which is hosted in AWS. The problem we have been facing is that current solution results in a lot of data transfer b/w instances across different AWS availability-zones which contributes to a big chunk of cost.

As for the issue with shard data rate limit, I don't think this change would have any impact. Mapping of traceIds to specific shard should be random enough to still have even distribution of records across shards. But will keep an eye out for any issues with hot partitioning.

Created a pull request to add this feature #33563

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awskinesis: @Aneurysm9 @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awskinesis: @Aneurysm9 @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.