Remove queue from the Prometheus remote write

[rakyll]

Prometheus remote write (PWR) exporter is supporting queued retry
out of the box, but due to the strict requirements of PWR, the queue
causes "out of order sample" errors. The queue won't be suitable
for Prometheus unless it has a way to shard the data by timeseries.

This is not going to break the existing users because if they have
this feature enabled, they already can't use the PRW exporter correctly.

Fixes #2949.

[codecov]

Codecov Report

Merging #2951 (9d12db2) into main (e72553a) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2951      +/-   ##
==========================================
- Coverage   91.68%   91.68%   -0.01%     
==========================================
  Files         312      312              
  Lines       15339    15337       -2     
==========================================
- Hits        14063    14061       -2     
  Misses        870      870              
  Partials      406      406              

Impacted Files	Coverage Δ
exporter/prometheusremotewriteexporter/factory.go	100.00% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e72553a...9d12db2. Read the comment docs.

[rakyll]

cc @odeke-em

[odeke-em]

LGTM, thank you @rakyll! I added a suggestion but also would be nice to add the issue you filed, as a reference to the code comments. Thank you.

[odeke-em]

Kindly cc-ing @bogdandrutu!

[rakyll]

Added a link to the issue, thanks for the suggestion @odeke-em.

[odeke-em]

Nice, thank you @rakyll! LGTM.

[tigrannajaryan]

Can we instead force queued_retry to have a single consumer for this exporter? That should fix the problem while keeping the retrying and queuing capabilities, right? I am not sure how important is that for this exporter though.

[rakyll]

@tigrannajaryan Retrying is critical in the context of remote write, it's not easy to have a highly available Prometheus, Cortex or Thanos. We want to reimplement the retrying options Prometheus server implements natively (https://prometheus.io/docs/practices/remote_write/) to provide a drop-in replacement and help users to reuse their fine tuned configuration.

[tigrannajaryan]

@tigrannajaryan Retrying is critical in the context of remote write, it's not easy to have a highly available Prometheus, Cortex or Thanos. We want to reimplement the retrying options Prometheus server implements natively (https://prometheus.io/docs/practices/remote_write/) to provide a drop-in replacement and help users to reuse their fine tuned configuration.

Sounds good. Do you also want to add a TODO or an issue (if it does not already exist) describing what you plan to do?

[rakyll]

I updated https://github.com/open-telemetry/opentelemetry-collector/issues/2259 to make sure we're following up with the necessary changes. Thanks much.