Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sending a random routing key for each message in kafka output #4573

Closed
danielnelson opened this issue Aug 20, 2018 · 0 comments · Fixed by #4579
Closed

Support sending a random routing key for each message in kafka output #4573

danielnelson opened this issue Aug 20, 2018 · 0 comments · Fixed by #4579
Labels
area/kafka feature request Requests for new plugin and for new features to existing plugins
Milestone

Comments

@danielnelson
Copy link
Contributor

Feature Request

Proposal:

Kafka requires at least one partition per concurrent consumer, and without a properly set routing key (aka partition key) only one partition will be written too. There is some information suggesting that the partition written to will be changed after 10m, but this would be too long of a windows for most uses.

While sometimes it is possible to use a common tag such as host, and in many respects this is ideal as it maintains ordering of each series, this tag is not always set.

Add a new parameter to the Kafka output plugin to add a random routing key to each message.

Current behavior:

Messages can be sent without a routing key or using the routing_tag option to choose a tag whose value is used as the routing key.

Desired behavior:

Similar to the amqp output, add a new routing_key option to set a static key, or if set to "random" then generate a random key.

[[outputs.kafka]]
  routing_key = "random"

Use case:

Without a common tag between all metrics this is required for spreading writes to multiple partitions.

@danielnelson danielnelson added feature request Requests for new plugin and for new features to existing plugins area/kafka labels Aug 20, 2018
@danielnelson danielnelson added this to the 1.8.0 milestone Aug 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kafka feature request Requests for new plugin and for new features to existing plugins
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant