Support ingesting exemplars into TSDB when blocks storage is enabled (cortexproject#4124)

* Add experimental support for ingesting exemplars from remote write into tsdb when blocks storage is enabled.

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Comment for clarity

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Skip entire exemplar loop if series ref missing

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplar metrics for distributor received/in and validation discarded per reason

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Fix spacing

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Comments

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Don't alloc empty sample slice when validating series with only exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Track exemplars that can't be ingested due to missing ref as failed

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update ingester tests for new exemplar metric names

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* make doc for new max_exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Update distributor validation messages and comments based on feedback. Count runes in exemplar labels instead of bytes

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Convert some exemplar metrics to global instead of per-user based on feedback about limiting cardinality. Update other code formatting and logic based on review feedback

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Update pkg/util/validation/errors.go

Simplify string formatting

Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add test cases based on review comments

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Update changelog

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Peter Štibraný <peter.stibrany@grafana.com>

Author: 3 people

CHANGELOG.md

@@ -15,6 +15,7 @@
   * `cortex_alertmanager_state_initial_sync_duration_seconds`
   * `cortex_alertmanager_state_persist_total`
   * `cortex_alertmanager_state_persist_failed_total`
+* [ENHANCEMENT] Blocks storage: support ingesting exemplars.  Enabled by setting new CLI flag `-blocks-storage.tsdb.max-exemplars=<n>` or config option `blocks_storage.tsdb.max_exemplars` to positive value. #4124
 * [BUGFIX] Purger: fix `Invalid null value in condition for column range` caused by `nil` value in range for WriteBatch query. #4128
 
 ## 1.9.0 in progress

docs/blocks-storage/querier.md

@@ -806,4 +806,9 @@ blocks_storage:
     # limit the number of concurrently opening TSDB's on startup
     # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup
     [max_tsdb_opening_concurrency_on_startup: <int> | default = 10]
+
+    # Enables support for exemplars in TSDB and sets the maximum number that
+    # will be stored. 0 or less means disabled.
+    # CLI flag: -blocks-storage.tsdb.max-exemplars
+    [max_exemplars: <int> | default = 0]
 ```

docs/blocks-storage/store-gateway.md

@@ -852,4 +852,9 @@ blocks_storage:
     # limit the number of concurrently opening TSDB's on startup
     # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup
     [max_tsdb_opening_concurrency_on_startup: <int> | default = 10]
+
+    # Enables support for exemplars in TSDB and sets the maximum number that
+    # will be stored. 0 or less means disabled.
+    # CLI flag: -blocks-storage.tsdb.max-exemplars
+    [max_exemplars: <int> | default = 0]
 ```

docs/configuration/config-file-reference.md

@@ -4943,6 +4943,11 @@ tsdb:
   # limit the number of concurrently opening TSDB's on startup
   # CLI flag: -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup
   [max_tsdb_opening_concurrency_on_startup: <int> | default = 10]
+
+  # Enables support for exemplars in TSDB and sets the maximum number that will
+  # be stored. 0 or less means disabled.
+  # CLI flag: -blocks-storage.tsdb.max-exemplars
+  [max_exemplars: <int> | default = 0]
 ```
 
 ### `compactor_config`

docs/configuration/v1-guarantees.md

@@ -70,3 +70,4 @@ Currently experimental features are:
   - `-ingester.stream-chunks-when-using-blocks` CLI flag
   - `-ingester_stream_chunks_when_using_blocks` (boolean) field in runtime config file
 - Instance limits in ingester and distributor
+- Exemplar storage, currently in-memory only within the Ingester based on Prometheus exemplar storage (`-blocks-storage.tsdb.max-exemplars`)