(otelarrowexporter) README: on batching w/ otel-arrow #35225
Conversation
|
Document change only, please merge. |
| To configure an OpenTelemetry Collector pipeline for both batching and | ||
| back-pressure, use of a custom component, the Concurrent Batch Processor, | ||
| available in the OTel-Arrow project repository, is required. We have not | ||
| included this in the Collector-Contrib repository because equivalent | ||
| functionality is being added as a standard exporter-batcher mechanism and the | ||
| new exporter-batcher functionality is still experimental. |
There was a problem hiding this comment.
As a user, how would I configure this to be include this into my collector?
Would it be possible add like a otb (collector builder) example? Or a link earlier in this text to go follow this and learn more there?
My concern that this is a lot of "expert" mode configuration with no warning that it is.
There was a problem hiding this comment.
This is a response to the reality today, which is that the none of the built-in support actually helps with the problem I am trying to solve--which is to have a synchronous pipeline with error transmission that is not limited to one export at a time. I am desperate to fix these problems! See the issue:
open-telemetry/opentelemetry-collector#11308
If you feel that this "expert-mode" is really a problem, then we should address the underlying issue--I ask you to approve open-telemetry/opentelemetry-collector#11324. If you do not feel that there is a problem, then let's merge this PR.
|
Again, this document explains the reality of the situation. If we can't improve batching processes, at least we can document how to work around the problem. I would still like this to merge. |
Description: Add documentation on the recommended methods for batching in conjunction with the OTel-Arrow exporter.
Testing: Advice based on performance experiments summarized in open-telemetry/opentelemetry.io#5198.
Documentation: This.