Prometheus remote write exporter implement v2

[jmichalek132]

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

Prometheus remote write v2 spec is close to being finalized, it would be good time to start working on implementing support for it in the prometheus remote write exporter component. Soon backend such as prometheus, mimir, thanos etc will start supporting it. It would great to start working on implementing to gain it's benefits,.

Describe the solution you'd like

I would like to contribute implementation of the remote write v2 prometheus/docs#2462 into the prometheus remote write exporter, most likely behind a feature flag since long-term v2 version is supposed to replace the current implementation.

Describe alternatives you've considered

Potentially it could introduced as a separate component but I hope that won't be necessary.

Additional context

The current implementation of remote write v2 lives in it's own branch of prometheus, https://github.com/prometheus/prometheus/tree/remote-write-2.0 , but should be merged soon.

TODO

Translation layer:

• Support for metadata
• Handle conflicts in metrics
• Handle target_info
• Handle exemplars
• Support for metric types
  • MetricTypeSum
  • MetricTypeHistogram
  • MetricTypeExponentialHistogram
  • MetricTypeSummary

Exporter:

• Implement batching
• Sort timeseries [exporter/prometheusremotewrite] feat: prom rw exporter add support for rw2 #35888 (comment)
• Add test case with invalid proto message [exporter/prometheusremotewrite] feat: prom rw exporter add support for rw2 #35888 (comment)
• Go over the spec and confirm it follows it
• Optionally add e2e tests?

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for the proposal @jmichalek132, makes sense to me. Agreed that a feature gate is a good idea here. 👍

[varshith257]

Interesting! How can I contribute to this feature? @crobert-1 ,@jmichalek132

[rapphil]

Remote write 2.0 is still experimental in prometheus, however it will be enabled by default in the the upcoming version version 2.54.0.

Having said that I would welcome a contribution that will add support to PRW2.0. I think we can add a configuration option protocol_version that can be used by users to select what version of the remote write protocol they want to use. Valid values would be 1.0 and 2.0 with default value 1.0. Once Prometheus deprecate 1.0 we can plan on moving the default to 2.0.

[varshith257]

Thanks @rapphil . Will look into the Prometheus remote write 2.0 and plan the changes with above said

[dashpole]

cc @ArthurSens
We were considering making this a CNCF intern project: cncf/mentoring#1281

[varshith257]

@dashpole Good to hear that this issue under CNCF Mentorship. Would like to contribute to this under LFX Mentorship.

What could be application process to be part of applying to this project?
cc: @ArthurSens

[ArthurSens]

Hi @varshith257, the application happens through the LFX website. There's a lot of instructions on their website :)

https://docs.linuxfoundation.org/lfx/mentorship/mentees

[varshith257]

@ArthurSens Thanks for pointing me there :)

Right now will apply to this project. Any other tasks that has to be done before application apart from resume and cover letter?

[ArthurSens]

@ArthurSens Thanks for pointing me there :)

Right now will apply to this project. Any other tasks that has to be done before application apart from resume and cover letter?

I don't think there's any other requirements :)

[manulpatel]

Hello! @ArthurSens @dashpole @aknuds1
I am Manul, backend Software engineer from India. I am interested to contribute in the development of RWv2 exporter and have applied under LFX mentorship 2024. I am trying to get myself familar with the Prometheus and OTEL through docs. I have successfully setup Prometheus locally for development.
While going through the RW2.0 specs, I learnt about the new Protobuf message specs io.prometheus.write.v2.Request. Also in the v2 specs sheet, multiple references are given to v1 docs to compare and explain repetitive reasons in FAQs, but it seems the link is not working or the docs are moved. So is it necessary to go through it?

Apart from these, what do you suggest to try on to learn/test, relavant to this feature and also more generic to get involved with active code contributions in both the organizations.

[dashpole]

I think it should link to https://prometheus.io/docs/specs/remote_write_spec/. I would try out the existing PRW exporter with the prometheus server as a first step. There should be time to get familiar with the components after the internship starts if you are selected.

[ArthurSens]

more generic to get involved with active code contributions in both organizations.

Most open-source projects have community meetings and guidelines somewhere in their repo:

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md https://github.com/prometheus/prometheus/blob/main/CONTRIBUTING.md

These documents usually guide you in a good direction for code contributions :)

It's also a good idea to look for issues labeled as good first issue or help wanted. Communicating over slack (in the appropriate channels) usually speeds up the process as well, just keep in mind that we're all volunteers here and response times might not be great sometimes.

[jmichalek132]

Ping @dashpole could you please assign this to me as I was selected for the lfx mentorship project?
Thank you.