-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New component: GRPC processor/connector #20888
Comments
Related: #11772 |
Just as a source of reference, as you may know, jaeger provides one feature to extend storage backend using a https://github.com/jaegertracing/jaeger/blob/main/plugin/storage/grpc/README.md |
@yurishkuro, if you could, would you please share your experience in supporting gRPC plugins, as well as the observed performance implications? My impression is that it's a good enabler, a good hack, but not something folks would use for serious production cases. |
I'm not sure I follow - can we just use the OTLP gRPC exporter to export to a gRPC server? |
We can use OTLP gRPC exporter and receiver to do this, but in my view, this approach has two down sides. 1 - Collector configuration can start to be a mess because you need to have multiple receivers and exporters to avoid since you can't reuse the existing receiver. Imagine I have a log receiver and I want to use an external server to enrich the log with some useful information, so I can use the OTLP exporter to send it to the external service, and then the external service can send it to the collector again, but I can't receive the log entry back on the log receiver, because this may lead on a loop this will require me to add another receiver to get the data back and this will require users open more ports on their collector. 2 - Using the receiver and exporter and not using the processor/connector seems to me that I'm not handling the data through the pipeline, maybe this can be just a semantic view I have, but looks better if we have a clear flow like Receiver > Process > Export |
@atoulme, I understood the proposal as to building a gRPC processor that would allow people to build gRPC plugins, allowing them to build custom processors and host them in external binaries, likely using https://github.com/hashicorp/go-plugin . |
I would avoid hashicorp plug-in lib for this. It's not significantly different from a plain grpc server, but imposes a sidecar pattern that it manages. There are other ways of running something as a sidecar. Having just a plain grpc interface decouples this from the sidecar consideration. |
We discussed this during the SIG meeting. Apologies if the conversation was hard to follow @nicolastakashi as I understood you had audio issues. If you watch the recording, there might be more context attached to the discussion you might see. The folks present in the discussion explained that to them this seemed like a departure from the model followed by the pipeline, where processor would typically be running in process, avoiding to make synchronous calls to external, I/O or network bound processes or devices. The reason is to avoid having to add complexity from working with processors. We try to organize around the lifecycle of the pipeline execution by applying backpressure or retries or other scaling mechanisms at that level. The same folks expressed that the way to organize what you envision, in their view, consists in using a OTLP exporter and a separate pipeline with a OTLP receiver. This allows asynchronous behaviors and doesn't require implementation work. Please comment here or feel free to participate in the SIG meetings (we have several ones with different time zones now) if you'd like to follow up, and thank you for putting this idea together. |
Just to clarify my understanding. I think the OP is proposing this: graph LR
R1[Receiver] --> |"(1)"| P1[Processor]
P1 --> |"(2) Remote gRPC"| P2
P2 --> |"(3) resp"| P1
P1 --> |"(4)"| E(Exporter)
subgraph Main OTEL Collector
R1
P1
E
end
subgraph Custom Server
P2(Plugin Processor)
end
@atoulme's response suggests putting the custom logic in the "Plugin Server" after the main exporter and doing whatever is needed there: graph LR
R1 --> P1
P1 --> E1
E1 --> R2
R2 --> P2
subgraph Main OTEL Collector
R1[Receiver]
P1[Processor]
E1(Exporter)
end
subgraph Custom Server
R2[Receiver]
P2[Custom handling]
end
The problem with the latter is that if OP wants the custom logic to be implemented in a different language they have to throw away all of the OTEL Collector's standard processing & export infrastructure and roll out their own. Whereas all they way is to introduce an additional enrichment step into the pipeline. FWIW, the blocking IO concerns inside the main processing pipeline are certainly valid, if the pipeline was not initially designed with that in mind. |
Thanks for the graphs. This is what I'm describing: graph TD
R1 --> P1
P1 --> E1
E1 --> R2
R2 --> P2
P2 --> E2
E2 --> R3
R3 --> P3
P3 --> E3
subgraph Main OTEL Collector
R1[Receiver]
P1[Processor]
E1(OTLP Exporter)
R3[OTLP Receiver]
P3[Processor]
E3(Exporter)
end
subgraph Custom Server
R2[OTLP Receiver]
P2[Custom handling]
E2[OTLP Exporter]
end
|
I think OP already stated that this is not the solution they are looking for. If they had the option of coding custom processing in Go, then why even bother with a separate process. |
I am just relaying the discussion from the meeting.
OP mentioned this in their motivation: it is to avoid building their own distribution of the collector. |
Hey folks! After some days thinking about that, I do agree with you that including asynchronous calls to the collector pipeline, may require more effort in terms of operations such as state management, to be honest as I said is a fair argument to not include this kind of integration on the collector. On the other hand, I would like to highlight some other points: 1 - As @yurishkuro mentioned, this kind of integration may let people from other community languages extend the OpenTelemetry Collector without knowing how to code Go. 2 - We open a good door to the community to build their own processors that may be useful for other contexts but not be useful to be available on this repository and track this kind of processor we may build something like awesome-otel-collector. Again I do understand the decision the working group thinks it's not a use case that OTEL wants to handle, but I think it brings some nice opportunities. |
Backing on this folks! |
I brought this up again during the SIG meeting on 7/12 as a discussion item regarding open-telemetry/opentelemetry-collector#7961 |
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
This issue has been closed as inactive because it has been stale for 120 days with no activity. |
The purpose and use-cases of the new component
The OpenTelemetryContrib is providing an amazing job extending the Otel collector capacities and also providing basic functionalities that might be useful for the entire community.
Although many other engineers would love to extend the OpenTelemetry Collector pipeline without the need to maintain their own OpenTelemetry distribution, which requires some work to keep track of the upstream changes.
With that in mind would be amazing if there's a generic processor to send the telemetry data to a GRPC server, so the engineers can build their own GRPC processors/connectors to manipulate the telemetry data and reply back to the collector, all this integration should be based on the OTEL protocol and a well-defined contract.
This kind of functionality may improve the user experience to extend the OpenTelemetry collector and also provide integrations to the external systems to enrich the telemetry data.
Example configuration for the component
Telemetry data types supported
Is this a vendor-specific component?
Sponsor (optional)
No response
Additional context
We had a short initial conversation on the CNCF Slack channel about this, as you can see here.
The text was updated successfully, but these errors were encountered: