[pulsarreceiver] - Receive logs in normal JSON or plain text from pulsar topics

[VihasMakwana]

Component(s)

No response

Is your feature request related to a problem? Please describe.

We can send raw strings or JSON format messages to the pulsar topic.
Currently, the pulsar receiver only expects otlp format for logs and metrics.

Describe the solution you'd like

I'd like to add a string and JSON unmarshler.

Describe alternatives you've considered

No alternative, I will work on a PR.

Additional context

No response

[github-actions]

Pinging code owners:

• pulsarreceiver: @dmitryax @dao-jun

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

Pinging code owners for receiver/pulsar: @dmitryax @dao-jun. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

I don't have a lot of context here, so I'm just trying to get an understanding of the purpose of the request. It looks like Pulsar publishes messages to topics, and it's always in the same format.

Your request is that once the data is received by the collector it should be able to be unmarshalled (decoded) into raw text or JSON. Is that correct? Do you have a specific use-case where this is really helpful? I'm wondering what the shortcoming here is of the OTLP format.

[VihasMakwana]

Yes, you're spot on.
Many tools/cloud services can use text/json format for sending logs.
We had a similar request for kafka receiver as well #20734

And we're facing similar thing with pulsar.

[VihasMakwana]

I'm currently working on getting a PR up.

[crobert-1]

I'll defer to the code owners if they have any thoughts otherwise, but sounds good to me. Thanks for your help here!

[dmitryax]

We recently started working on a new way to specify encoding for receiver/exporters. I believe this is a good opportunity to test it in real life instead of introducing the encoding embedded into the receiver which will be replaced at some point. We already have text encoder, see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding. We need to introduce a json encoding extension and add ability to use the encoding extensions in the pulsar receiver

[VihasMakwana]

@dmitryax sounds great. Will work on it.

[dao-jun]

We recently started working on a new way to specify encoding for receiver/exporters. I believe this is a good opportunity to test it in real life instead of introducing the encoding embedded into the receiver which will be replaced at some point. We already have text encoder, see https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding. We need to introduce a json encoding extension and add ability to use the encoding extensions in the pulsar receiver

Great! So we can move the encoding/decoding logics from receiver/exporter to extensions and use configuration item to specify which encoding/decoding to use