Prometheus Remote Write v2 Implementation

[SungJin1212]

I'm implementing Prometheus remote write v2 (PRW2.0) based on PRW2.0 docs (https://prometheus.io/docs/specs/remote_write_spec_2_0).

There are two milestones:

Milestone1

The distributor can accept PRW2.0 requests. It requires small changes between the Distributor and Ingester, including adding response headers. The ingestion is done by converting the PRW2.0 request to PRW1.0, then using existing Push methods in the Distributor and Ingester.

push response headers to be added:

X-Prometheus-Remote-Write-Samples-Written <count of all successfully written Samples>
X-Prometheus-Remote-Write-Histograms-Written <count of all successfully written Histogram samples>
X-Prometheus-Remote-Write-Exemplars-Written <count of all successfully written Exemplars>

• Support remote write v2 by converting request #6330
• Allow NHCB after Add native histogram custom values #6687 merged
• Introduce sync.pool to avoid lots of GC

Milestone2

Add PRW2.0 proto and update the protocol used between the Distributor and Ingester.

• Add RPW2.0 proto
• Implement Distributor & Ingester Push functions
• Add fallback logic for rolling updates: when the Distributor can accept PRW2.0 but not Ingester, we have to ingest metrics.
• Add some performance tests between PRW2.0 and 1.0.

[yeya24]

Thanks!
I think we should start with milestone 1 and having milestone 1 is good enough to support RW 2.0

We need to think more about milestone 2 before actually changing the internal write protocol between distributor and ingester. They are not user facing and we can do whatever we want so doesn't need to be same as RW 2.0

[SungJin1212]

We can test it using the client-golang library. The upstream client-golang pr is here: prometheus/client_golang#1658.