-
Couldn't load subscription status.
- Fork 1.9k
enhancement(aggregate): Add Distribution as an aggregation mode #23994
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
enhancement(aggregate): Add Distribution as an aggregation mode #23994
Conversation
…/jlambatl/vector into transform_aggregate_distribution
…improve the documentation that is generated for the distribution aggregation mode in the configuration parameter
|
I've pushed up a commit that should resolve the comments/feedback provided. It's hard to find the balance between being too verbose and not providing enough clarity. |
|
Starting to look good, will wait for a review from @vectordotdev/documentation. |
| }) | ||
| .await; | ||
| } | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @jlambatl, I fixed some tests and added new ones. Can you a thorough review of the latest code? I think this PR is almost ready to merge.
Summary
This change adds a new AggregateMode
distributionthat aggregates individual metrics into a distribution. This doesn't reduce the accuracy of the metrics; instead, it reduces the volume of payloads being emitted from Vector by reducing the overhead of duplicate tags and other fields in exchange for a single larger payload.This pattern suits real-time analytics databases, where histograms and other statistical functions can be calculated on the fly. It reduces the need to manually codify histogram buckets, which can become an onerous task on systems dealing with large metric volumes.
The distribution is sorted in the current implementation. I have been unsure whether that is a good idea, but I'm happy to change that based on your feedback.
closes: #24001
Vector configuration
How did you test this PR?
I have added many different unit tests that cover the primary aggregation types, histograms, distributions, etc.
Change Type
Is this a breaking change?
Does this PR include user-facing changes?
no-changeloglabel to this PR.Notes
N/A