Partitioning/batching based on instrumentation scope #23848
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Use Cases
This is a feature request to improve the batching and performance of the opentelemetry sink.
Currently the opentelemetry sink uses the http sink implementation and thus it uses the same partitioning strategy, which uses the uri and the headers as the key.
Note 2: Here we examined a logs use case, but the same applies to metrics and traces.
Attempted Solutions
Note: ResourceLogs can contain multiple ScopeLogs and each ScopeLogs instance can contain many LogRecords. This is another grouping layer that can be leveraged today to group OTEL logs.
Proposal
A natural way to group multiple ResourceLogs requests is to use InstrumentationScope as the key. Introduce new configuration option to support this new grouping strategy.
References
Vector event batching is broken for this sink: #22054 (comment)
Version
No response