[Prometheus remote-write receiver] Create a receiver to send metrics using Prometheus remote-write API

[carlosjgp]

Is your feature request related to a problem? Please describe.
We are trying to design an architecture where the metrics, traces and logs collectors have as a little CPU and memory footprint as possible on the K8s clusters where we deploy them

Cluster client                                                      Centralised receivers
--------------------------------------------|                      |---------------------------------------------|
Prometheus -- remote-write--->| OTEL agent--|-----otlp mTLS------> |---> OTEL deployment--->|---> Thanos receive |
Apps ----------otel traces--->|             |                      |                        |---> Grafana Tempo  |
Promtail--------logs--------->|             |                      |                        |---> Grafana Loki   |
--------------------------------------------|                      |---------------------------------------------|

Describe the solution you'd like
Create a receiver that is able to expose a port compatible with Prometheus Remote writer API so I can still use Prometheus capabilities to collect metrics and OTEL agent to receive traces, decorate (processors) and route them to the correct tracing, metrics and logging storage

Describe alternatives you've considered
Use OTEL agent to collect metrics as a Prometheus replacement but Rules are not part of the integration and I also use PrometheusOperator, which seems to have a receiver proposal but still no Rules support.

Prometheus Rules are evaluated per cluster and sent to the centralised cluster where Thanos Ruler will evaluate them and contact AlertManager

For now, I'll send the metrics directly from Prometheus to Thanos receive over HTTPS with TLS auth

Additional context

[dmitryax]

cc @Aneurysm9

[dmitryax]

We discussed adding prometheus remote write capabilities to the receiver on the SIG meeting today. Everyone agrees that we need it. It should be implemented as another receiver called prometheusremotewrite. No one of the active OTel contributors has plans to work on it at this point. So any help would be appreciated. cc @bogdandrutu @Aneurysm9

[nlamirault]

Nice feature ! With this one we could use the whole Prometheus Operator stack and OpenTelemetry together.

[dmitryax]

We are still looking for contributors to implement this. @nlamirault @carlosjgp let us know if you are interested in doing it

[arajkumar]

@dmitryax IIUC, the following works has to be done for this,

1. Add translator for prompb.TimeSeries => pmetric.Metrics conversion
2. Add a receiver/prometheusremotewrite

May be I can give a try if nobody is actively working on this.

[jpkrohling]

@arajkumar, @kovrus will work on this one, but your help is appreciated I reviewing the PR once it's ready and in trying it out.