store: add streamed snappy encoding for postings list #6303

GiedriusS · 2023-04-20T13:03:08Z

We've noticed that decoding Snappy compressed postings list takes a lot of RAM:

(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
      flat  flat%   sum%        cum   cum%
         0     0%     0%  1905.67GB 90.20%  golang.org/x/sync/errgroup.(*Group).Go.func1
    2.08GB 0.098% 0.098%  1456.94GB 68.96%  github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
    1.64GB 0.078%  0.18%  1454.87GB 68.86%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
    2.31GB  0.11%  0.29%  1258.15GB 59.55%  github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
    1.48GB  0.07%  0.36%  1219.67GB 57.73%  github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
 1215.21GB 57.52% 57.87%  1215.21GB 57.52%  github.com/klauspost/compress/s2.Decode

This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that.

Notice that our current index.Postings list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward.

Benchmark data:

name                                                          time/op
PostingsEncodingDecoding/10000/raw/encode-16                  71.0µs ± 6%
PostingsEncodingDecoding/10000/raw/decode-16                  93.1ns ± 9%
PostingsEncodingDecoding/10000#01/snappy/encode-16            72.6µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16            1.78µs ±14%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16    91.0µs ± 1%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16    15.1µs ±19%
PostingsEncodingDecoding/100000/raw/encode-16                  702µs ± 1%
PostingsEncodingDecoding/100000/raw/decode-16                 91.6ns ±10%
PostingsEncodingDecoding/100000#01/snappy/encode-16            717µs ± 1%
PostingsEncodingDecoding/100000#01/snappy/decode-16           16.2µs ±25%
PostingsEncodingDecoding/100000#02/snappyStreamed/encode-16    890µs ± 1%
PostingsEncodingDecoding/100000#02/snappyStreamed/decode-16   14.7µs ± 9%
PostingsEncodingDecoding/1000000/snappy/encode-16             7.47ms ± 3%
PostingsEncodingDecoding/1000000/snappy/decode-16              157µs ±12%
PostingsEncodingDecoding/1000000#01/snappyStreamed/encode-16  9.20ms ± 4%
PostingsEncodingDecoding/1000000#01/snappyStreamed/decode-16  15.0µs ± 7%
PostingsEncodingDecoding/1000000#02/raw/encode-16             7.24ms ± 3%
PostingsEncodingDecoding/1000000#02/raw/decode-16             82.3ns ± 5%

name                                                          alloc/op
PostingsEncodingDecoding/10000/raw/encode-16                  13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                   96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16            25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16            11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16    16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16     148kB ± 0%
PostingsEncodingDecoding/100000/raw/encode-16                  131kB ± 0%
PostingsEncodingDecoding/100000/raw/decode-16                  96.0B ± 0%
PostingsEncodingDecoding/100000#01/snappy/encode-16            254kB ± 0%
PostingsEncodingDecoding/100000#01/snappy/decode-16            107kB ± 0%
PostingsEncodingDecoding/100000#02/snappyStreamed/encode-16    149kB ± 0%
PostingsEncodingDecoding/100000#02/snappyStreamed/decode-16    148kB ± 0%
PostingsEncodingDecoding/1000000/snappy/encode-16             2.48MB ± 0%
PostingsEncodingDecoding/1000000/snappy/decode-16             1.05MB ± 0%
PostingsEncodingDecoding/1000000#01/snappyStreamed/encode-16  1.47MB ± 0%
PostingsEncodingDecoding/1000000#01/snappyStreamed/decode-16   148kB ± 0%
PostingsEncodingDecoding/1000000#02/raw/encode-16             1.25MB ± 0%
PostingsEncodingDecoding/1000000#02/raw/decode-16              96.0B ± 0%

name                                                          allocs/op
PostingsEncodingDecoding/10000/raw/encode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16                    2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16              3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16              4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16      4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16      5.00 ± 0%
PostingsEncodingDecoding/100000/raw/encode-16                   2.00 ± 0%
PostingsEncodingDecoding/100000/raw/decode-16                   2.00 ± 0%
PostingsEncodingDecoding/100000#01/snappy/encode-16             3.00 ± 0%
PostingsEncodingDecoding/100000#01/snappy/decode-16             4.00 ± 0%
PostingsEncodingDecoding/100000#02/snappyStreamed/encode-16     4.00 ± 0%
PostingsEncodingDecoding/100000#02/snappyStreamed/decode-16     5.00 ± 0%
PostingsEncodingDecoding/1000000/snappy/encode-16               3.00 ± 0%
PostingsEncodingDecoding/1000000/snappy/decode-16               4.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappyStreamed/encode-16    4.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappyStreamed/decode-16    5.00 ± 0%
PostingsEncodingDecoding/1000000#02/raw/encode-16               2.00 ± 0%
PostingsEncodingDecoding/1000000#02/raw/decode-16               2.00 ± 0%

Compression ratios are still the same like previously:

$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN   TestDiffVarintCodec/snappy/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44498 bytes
    postings_codec_test.go:81: ratio: 0.012
=== RUN   TestDiffVarintCodec/snappyStreamed/i!~"2.*"
    postings_codec_test.go:73: postings entries: 944450
    postings_codec_test.go:74: original size (4*entries): 3777800 bytes
    postings_codec_test.go:80: encoded size 44670 bytes
    postings_codec_test.go:81: ratio: 0.012

I found s2 writer/reader to have worse performance so I opted for the vanilla snappy encoder/decoder.

We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

pjarmalavicius

👏

GiedriusS · 2023-04-20T14:21:07Z

I don't see the decoding function in profiles anymore after this is deployed in sandbox.

yeya24

Thanks. I will also take a look at this today.

GiedriusS · 2023-04-25T09:57:15Z

Here's RAM usage after deploying this + clearing out old keys in Redis in prod on a server that only runs Thanos Store. Similar story on other servers.
@yeya24 maybe you've had some time to look at this?

pkg/store/postings_codec.go

matej-g

This looks good to me as far as I can follow the en/decoding logic 👍. Couple clarifying questions from my side.

pkg/store/postings_codec.go

pkg/store/bucket.go

pkg/store/postings_codec.go

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

fpetkovski

Do we want to have a changelog entry for this? Also if the previous implementation was not released, maybe we don't we don't need to add a new encoding version.

fpetkovski · 2023-04-25T12:06:44Z

pkg/store/postings_codec.go

+}
+
+func diffVarintSnappyStreamedEncode(p index.Postings, length int) ([]byte, error) {
+	out := make([]byte, 0, estimateSnappyStreamSize(length))


How many of these allocations do we need to make, is it one per block?

Yes, perhaps we could improve this in the future. This part is unchanged from the previous encoding.

We could also pool this memory if needed. But since it's not showing on a profile, it might not have a big impact so it's fine to leave it for now.

fpetkovski · 2023-04-25T12:09:13Z

pkg/store/postings_codec.go

+
+	uvarintEncodeBuf := make([]byte, binary.MaxVarintLen64)
+
+	pooledSW := snappyWriterPool.Get()


Is there any way we could reuse the pool for grpc compression: https://github.com/thanos-io/thanos/blob/main/pkg/extgrpc/snappy/snappy.go?

Added this in 4a037bd, perhaps it's not too bad to call the same pool across packages?

I think the risk is that buffers for different use cases end up being vastly different in size, leading to some waste when resizing them. But I am not aware of any potential issues with correctness that could happen if we reuse the same pool from different packages.

Snappy writer uses the same buffer sizes for each writer instance so this should be a good change in that regard 👍

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

yeya24 · 2023-04-25T18:06:04Z

@GiedriusS Can you help me understand the memory usage graph? Did the memory usage increase after deploying the change? Sorry it is not very clear when you deployed this change

yeya24

But overall code changes look good!

GiedriusS · 2023-04-25T18:22:14Z

@GiedriusS Can you help me understand the memory usage graph? Did the memory usage increase after deploying the change? Sorry it is not very clear when you deployed this change

Store RAM usage is consistently smaller now. Another server where this change was deployed around 14:00:

fpetkovski

LGTM 🚀

We just need to resolve conflicts.

…ostings_decoding Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

fpetkovski

🚀

* mixins: Add code/grpc-code dimension to error widgets Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix messed up merge conflict resolution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Readd empty line at the end of changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * mixin(Rule): Add rule evaluation failures to the Rule dashboard (thanos-io#6244) * Improve Thanos Rule dashboard legends Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add evaluations failed to Rule dashboard Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor rule dashboard Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * added thanos logo in react app (thanos-io#6264) Signed-off-by: hackeramitkumar <amit9116260192@gmail.com> * Add an experimental flag to block samples with timestamp too far in the future (thanos-io#6195) * Add an experimental flag to block samples with timestamp too far in the future Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix bug Signed-off-by: Yi Jin <yi.jin@databricks.com> * address comments Signed-off-by: Yi Jin <yi.jin@databricks.com> * fix docs CI errors Signed-off-by: Yi Jin <yi.jin@databricks.com> * resolve merge conflicts Signed-off-by: Yi Jin <yi.jin@databricks.com> * resolve merge conflicts Signed-off-by: Yi Jin <yi.jin@databricks.com> * retrigger checks Signed-off-by: Yi Jin <yi.jin@databricks.com> --------- Signed-off-by: Yi Jin <yi.jin@databricks.com> * store/bucket: snappy-encoded postings reading improvements (thanos-io#6245) * store: pool input to snappy.Decode Pool input to snappy.Decode to avoid allocations. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use s2 for decoding snappy It's faster hence use it. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: small code style adjustment Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: call closefns before returning err Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: return both if possible Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/bucket: always call close fns Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * truncateExtLabels support Unicode cut (thanos-io#6267) * truncateExtLabels support Unicode cut Signed-off-by: mickeyzzc <mickey_zzc@163.com> * update TestTruncateExtLabels and pass test Signed-off-by: mickeyzzc <mickey_zzc@163.com> --------- Signed-off-by: mickeyzzc <mickey_zzc@163.com> * Update mentorship links Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix segfault in LabelValues during head compaction (thanos-io#6271) * Fix segfault in LabelValues during head compaction Head compaction causes blocks outside the retention period to get deleted. If there is an in-flight LabelValues request at the same time, deleting the block can cause the store proxy to panic since it loses access to the data. This commit fixes the issue by copying label values from TSDB stores before returning them to the store proxy. I thought about exposing a Close method on the TSDB store which the Proxy can call, but this will not eliminate cases where gRPC defers sending data over a channel using its queueing mechanism. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add changelog entry Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Assert no error when querying labels Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Mixin: Allow specifying an instance name filter (thanos-io#6273) This commit allow specifying the instance name filter, in order to filter the datasources shown on the dashboards. For example, when generating the dashboards one can do the following (i.e in config.libsonnet) ``` dashboard+:: { prefix: 'Thanos / ', ... instance_name_filter: '/EU.*/' ``` Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Adds Deno to adopters.yml (thanos-io#6275) Signed-off-by: Will (Newby) Atlas <will@deno.com> * Bump `make test` timeout (thanos-io#6276) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * fix 0.31 changelog (thanos-io#6278) Signed-off-by: junot <junotxiang@kubesphere.io> * Query: Switch Multiple Engines (thanos-io#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (thanos-io#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * tests: Remove custom Between test matcher (thanos-io#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * query frontend, query UI: Native histogram support (thanos-io#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (thanos-io#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (thanos-io#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixed modules, logicalplan flag and more * Made assets * Removed unused test function * Sorted labels --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Signed-off-by: hackeramitkumar <amit9116260192@gmail.com> Signed-off-by: Yi Jin <yi.jin@databricks.com> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: mickeyzzc <mickey_zzc@163.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> Signed-off-by: Will (Newby) Atlas <will@deno.com> Signed-off-by: junot <junotxiang@kubesphere.io> Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: naveadkazi <navead@carwale.com> Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Amit kumar <amit9116260192@gmail.com> Co-authored-by: Yi Jin <96499497+jnyi@users.noreply.github.com> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: MickeyZZC <mickeyzzc@gmail.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Jacob Baungård Hansen <jacobbaungard@redhat.com> Co-authored-by: Will (Newby) Atlas <willnewby@gmail.com> Co-authored-by: junot <49136171+junotx@users.noreply.github.com> Co-authored-by: Pradyumna Krishna <git@onpy.in> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Co-authored-by: Alex Le <emoc1989@gmail.com> Co-authored-by: naveadkazi <navead@carwale.com>

* store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with https://github.com/efficientgo/e2e/commit/c316eb95ae5b0fa65b0be110f8cfc3c48e8810bb. The added t.Logf() showed that is the problem is that with a lower bytes limit, it might hit the series or chunks part first. I have bumped the bytes limit. I calculated the new bytes limit by checking how much bytes are allocated before sending the last chunk. I have also noticed that one block is created without a delay. Update it so that it would be like the others. Include objstore@main update with https://github.com/thanos-io/objstore/pull/62/files so that Iter() would always return an error on a timeout. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * pkg/reloader: use watchInterval timeout for initial apply (#6519) * pkg/reloader: use watchInterval timeout for initial apply Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * changelog Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * use distinct context for initial sync Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> --------- Signed-off-by: Craig Peterson <192540+captncraig@users.noreply.github.com> * Standardize index cache metrics (#6523) * Add histogram metrics for index cache item size (#6528) * add histogram metrics for index cache item size Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Upgrade objstore (#6507) * Upgrade objstore Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update with the latest APIs Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> --------- Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Make compact lifecycle more flexible to be overridden for sharded compaction (#5964) * Updates busybox SHA (#6534) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * BucketedBytes to buffer byte slices when decoding postings from cache (#6531) * BucketedBytes to buffer byte slices when deconding postings from cache Signed-off-by: Alan Protasio <alanprot@gmail.com> * Fixing Lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Shipper: change upload compacted type from bool to a function (#6526) * change shipper upload compacted type from bool to a function Signed-off-by: Ben Ye <benye@amazon.com> * add default to false Signed-off-by: Ben Ye <benye@amazon.com> * reset uploadedCompacted to 0 Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Query: Forward tenant information via StoreAPI (#6530) * Querier: Forward tenant information downstream With this commit we attach tenant information to each query request and forward it via the StoreAPI to any downstream Store Gateways and Queriers. We add the following command lines options which mimics the tenant functionality in Receive. The options are currently hidden, as they provide no real functionality yet. This will come in future steps. --query.tenant-header --query.default-tenant --query.tenant-certificate Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Receive: Use CertificateField from Tenancy pkg These consts are now defined in the Tenancy package, so we should use those instead. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Deduplicate matchers in posting group (#6532) * deduplicate matchers in posting group Signed-off-by: Ben Ye <benye@amazon.com> * don't use map for deduplication Signed-off-by: Ben Ye <benye@amazon.com> * lint Signed-off-by: Ben Ye <benye@amazon.com> * address comments Signed-off-by: Ben Ye <benye@amazon.com> * fi…

* fix 0.31 changelog (#6278) Signed-off-by: junot <junotxiang@kubesphere.io> * Query: Switch Multiple Engines (#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <leqiyue@amazon.com> * Reverted test code Signed-off-by: Alex Le <leqiyue@amazon.com> * Updated comment Signed-off-by: Alex Le <leqiyue@amazon.com> * docs: mismatch in changelog Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * Updates busybox SHA (#6283) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> * trigger workflow Signed-off-by: Alex Le <leqiyue@amazon.com> --------- Signed-off-by: Alex Le <leqiyue@amazon.com> Signed-off-by: Etienne Martel <etienne.martel.7@gmail.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: Etienne Martel <etienne.martel.7@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <navead@carwale.com> * sort ProxyResponseHeap comparing only diff labels Signed-off-by: Walther Lee <walthere.lee@gmail.com> * proxy heap: remove store labels before comparing response series Signed-off-by: Walther Lee <walthere.lee@gmail.com> * tests: Remove custom Between test matcher (#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove testingRespSet and update comparison func Signed-off-by: Walther Lee <walthere.lee@gmail.com> * Fix tests Signed-off-by: Walther Lee <walthere.lee@gmail.com> * query frontend, query UI: Native histogram support (#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Add chnagelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Updated Readme Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> trigger tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Made assets Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * store: add streamed snappy encoding for postings list (#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: fix estimation Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: atomically replace no compact marked map (#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Reuse buffers for label comparison Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Use len instead of cap Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Fixed Jaeger exporter specification link (#6324) Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * receive: Make max backoff configurable (#6163) Signed-off-by: Philip Gough <philip.p.gough@gmail.com> * Replace summary in extprom metrics with histogram (#6327) * Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added changelog Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Expose info for each TSDB This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Add test case for proxy store Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Pass engine factory by reference Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Cleanup Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Use NewQueryEngineFactory in tests Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Avoid expensive log.Valuer evaluation for disallowed levels (#6322) Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com> * Fix inconsistent error for series limits in Store API (#6330) * store: fix inconsistent error for series limits Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update pkg/store/bucket.go Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * rename labelValues serires liimiter test function Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * *: remove unmaintained gzip library (#6332) Switch from nytimes gzip library to the klaustpost's gzip code. The old gzip HTTP handler shows up a lot in allocs so that's how I ended up doing this change. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Traces sampler env var (#6306) * Issue#5947 OTEL_TRACES_SAMPLER env var Signed-off-by: shayyxi <shazi12384@gmail.com> * Test correction Signed-off-by: shayyxi <shazi12384@gmail.com> * doc failure correction. parse float argument correction. Signed-off-by: shayyxi <shazi12384@gmail.com> * added the changelog. Signed-off-by: shayyxi <shazi12384@gmail.com> * ran make docs to fix the build failure. Signed-off-by: shayyxi <shazi12384@gmail.com> * corrected the incorrect change in tools.md Signed-off-by: shayyxi <shazi12384@gmail.com> * fixed review comments. Signed-off-by: shayyxi <shazi12384@gmail.com> --------- Signed-off-by: shayyxi <shazi12384@gmail.com> Signed-off-by: Shazi <42436533+shayyxi@users.noreply.github.com> Co-authored-by: shayyxi <shazi12384@gmail.com> * query: use storepb.SeriesServer (#6334) Use storepb.SeriesServer instead of the concrete struct. This allows implementing functionality on top of the proxy. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil: upgrade `rueidis` to v1.0.2 to improve error handling while shrinking a redis cluster. https://github.com/redis/rueidis/issues/209 (#6342) * use github.com/onsi/gomega/gleak to detect goroutine leak with timeout Signed-off-by: Rueian <rueiancsie@gmail.com> * fix: spelling errors DoInSpanWtihErr to DoInSpanWithErr (#6345) Signed-off-by: aimuz <mr.imuz@gmail.com> * Return grpc code resource exhausted for byte limit error (#6325) * return grpc code resource exhausted for byte limit error Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * update partial response strategy Signed-off-by: Ben Ye <benye@amazon.com> * fix limit Signed-off-by: Ben Ye <benye@amazon.com> * try to fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test error message Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Bump promql-engine to fix https://github.com/thanos-community/promql-engine/issues/239 (#6349) Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com> * Optimize generating store match reason (#6355) Removes reason generation when debug logging is not enabled. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Updates busybox SHA (#6365) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Query: Add +Inf bucket to query duration metrics (#6358) * Query: Add +Inf bucket to query duration metrics For the query duration metrics (`thanos_store_api_query_duration_seconds`), we record query respond latency, based on the size of the query (samples/series), and save to a histogram. However, when a query is made which exceeds the biggest sample/serie size, we would prior to this commit, put the request into the largest bucket. With this commit, we instead create an `+Inf` bucket, and put requests which are larger than the biggest defined bucket into that. This gives more accurate results, and also allow one to see if the bucket sizes are incorrectly sized. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Tests: Mutex around non-thread safe random source When creating test blocks, we use a non-thread safe random source, in multiple goroutines. Due to this, tests would sometime panic. This commits puts a mutex around calls using the same source, in order to avoid this. This should hopefully improve reliability of e2e tests. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> --------- Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Check context when expanding postings (#6363) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * ui: only keep name in store_matches param (#6371) We are doing store matching on the `name` field hence only keep that field in the URL because otherwise the URL could get quite lengthy with external labelsets inside of it. Besides unit tests, I have also tested locally: - Enable store filtering; - Select store(-s); - Copy/paste URL into the new tab and see that the same stores are loaded like expected; - See that URL only has names in them. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * docs: replace --store with --endpoint Replace deprecated `--store` with `--endpoint` in docs. Signed-off-by: Paul Gier <paul.gier@datastax.com> * Optimizing "grafana generated" regex matchers (#6376) * Opmizing Group Regex Signed-off-by: Alan Protasio <alanprot@gmail.com> * fixing native histogram tests Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cache: various index cache client improvements (#6374) * e2e(query): Reproduce dedup issue from #6257 Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add dedup e2e test for Receive With internal and external labels support. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Simplify generated blocks for query test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve query dedup test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Write a query test for dedup with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Refactor query dedup test with sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive query test Now it properly ensures the double dedup works (on internal and external labels). Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix receive drawing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add one extra test caes for query dedup from store Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Receive query with dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Complement test for Sidecar query dedup Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Query Explanation (#6346) * Return Query Explaination in QueryAPI A param `explain` is added to QueryAPI, if true then explanation returned by the `Explain()` method of the query having structure `ExplainOutputNode` is returned in response. Query Explanation is added under new field in response that is `thanosInfo`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add explain checkbox in thanos UI A explain checkbox is added to Thanos Query UI, that requests for query explanation from thanos query api. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add ExpandableNode Component ExpandableNode component renders Query Explanation in the thanos UI. Requires a new package `react-accessible-treeview`. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Disable Explain checkbox on prometheus engine Prometheus engine sends out error if toggle explain button. To provide better experience, the explain checkbox get disbaled on switching to prometheus engine and enable back on switching to thanos engine. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add alert box with horizontal scrolling for Explanation Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove ExpandableNode and Add ListTree Updates the design for query explanation box, removes `ExpandableNode` and the dependency. Builts a new `ListTree` that does the same using reactstrap and custom css. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor refactor in Query API response `thanosInfo` is removed from Query reponse and used `explanation` directly. `disableCheckbox` is also renamed to `disableExplainCheckbox` in thanos UI. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI tests to passing Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor UI changes and test fix UI improvements and Panel test fix other way around, resetting the results on panel construction. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update promql-engine to use Explain method Signed-off-by: Pradyumna Krishna <git@onpy.in> * Build UI assets Build UI assets, that runs new thanos UI with explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Revert proxy url change from package.json `proxy` was accidently changed and committed with package.json when removed dependency. Hence, reverting it back. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Minor changes in UI Fix requested changes in UI. - Rename `state` and `setState` to `mapping` and `setMapping`. - Rename `NodeTree` to `QueryTree`. - Use unicode characters instead of `-` and `+`. - Fix blue box on explain button. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Update UI assets Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Implementing Regex optimization on the `MatchNotRegexp` and `MatchNotEqual` matcher type (#6379) * Implementing Regex optimization on the MatchNotRegexp matcher type Signed-off-by: Alan Protasio <alanprot@gmail.com> * Opmizing MatchNotEqual Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Expected failure of block label query dedup tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back the correct makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * add de-cix as adopter (#6386) Signed-off-by: Raul Garcia Sanchez <info@raulgarcia.de> * Remove extra line that broke untouched test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add back line break at end of makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix Receive single ingestor test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup issue in Receive Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add even more test cases for dedup on store gw Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reproduce dedup bug in Sidecar Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reuse nginx image name Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Let all users read the metrics file from static metrics server Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * [chore] Updating Query Engine and Prometheus (#6392) * Updating Query Engine Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix prometheus breaking change Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update prometheus with https://github.com/prometheus/prometheus/pull/12387 Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Receive: Allow specifying tenant-specific external labels in RouterIngestor (#5777) Signed-off-by: haanhvu <haanh6594@gmail.com> * check context cancel when doing posting batches (#6396) Signed-off-by: Ben Ye <benye@amazon.com> * Expose store gateway query stats in series response hints (#6352) * expose query stats hints Signed-off-by: Ben Ye <benye@amazon.com> * update Signed-off-by: Ben Ye <benye@amazon.com> * add query stats hints in result Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * add merge method Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> modify hints proto Signed-off-by: Ben Ye <benye@amazon.com> fix unit test Signed-off-by: Ben Ye <benye@amazon.com> update format Signed-off-by: Ben Ye <benye@amazon.com> * update comments Signed-off-by: Ben Ye <benye@amazon.com> * try again Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * receive: make az aware ketama hashring (#6369) * receive: make az aware ketama hashring Signed-off-by: Alexander Rickardsson <alxric@aiven.io> * receive: pass endpoints in hashring config as object Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive: add some tests for consistent hashing in presence of AZs Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * receive,docs: add migration note for az aware hashring Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> --------- Signed-off-by: Alexander Rickardsson <alxric@aiven.io> Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> Co-authored-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Reformat asciiflow chart Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Proposal: query path tenancy (#6320) * Add 1st version of query path tenancy proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update proposal after initial feedback Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add cool picture Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Include example in cross tenant query complications Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve reasoning for why not using the QFE Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve writing in "How" section Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix owner profile link Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address few more PR review comments Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address feedback on flag name text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve non-goals text Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update diagram Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update docs/proposals-accepted/202304-query-path-tenancy.md Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify scenario for pitfalls of current solution Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Clarify that Store doesn't care about tenant label Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add an action plan Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention alternative idea of modifying Store API Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Address lots of feedback on the proposal Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Format query path tenancy proposal doc Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add a "Tenancy Model" subsection to "Goals" Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Mention header semanthics in comparison with gRPC message field Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve action plan structure and writing Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix double-counting bug in http_request_duration metric (#6399) * fix double-counting bug in http_request_duration metric Signed-off-by: 4orty <kwk5178@gmail.com> * Update Changelog Signed-off-by: 4orty <kwk5178@gmail.com> --------- Signed-off-by: 4orty <kwk5178@gmail.com> * Replace group with resolution in compact metrics. Compaction metrics have too high a cardinality, causing metric bloat on large installations. The group information is better suited to logs. * Replace with a `resolution` label to the compaction counters. Fixes: https://github.com/thanos-io/thanos/issues/5841 Signed-off-by: SuperQ <superq@gmail.com> * Reuse static metrics server from e2e framework Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run go mod tidy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Updates busybox SHA (#6403) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * Fix series stats merge (#6408) * fix series stats merge Signed-off-by: Ben Ye <benye@amazon.com> * update license header Signed-off-by: Ben Ye <benye@amazon.com> * use reflect Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: allow unlimited head_series_limit tenants (#6406) With this commit we now allow to configure tenants with unlimited active series limit by setting the limit to `0`. Prior to this commit setting a per tenant limit to `0` would cause the tenant to be unable to write any metrics at all. This fixes: https://github.com/thanos-io/thanos/issues/6393 Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * expose downloaded data size in query hints (#6409) Signed-off-by: Ben Ye <benye@amazon.com> * maintainers: add myself to triagers (#6414) Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Add `@douglascamata` to triagers (#6418) Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add Blog (#6411) * Add LFX blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Add Headers to blog Signed-off-by: Pradyumna Krishna <git@onpy.in> * Lint blog Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * blog: Fix images for LFX post (#6422) * blog: Fix images for LFX post Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * fix lint Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Index Cache: Change cache key for postings (#6405) * extend postings cache key with codec Signed-off-by: Ben Ye <benye@amazon.com> * add changelog Signed-off-by: Ben Ye <benye@amazon.com> * update code back Signed-off-by: Ben Ye <benye@amazon.com> * add colon Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix another test Signed-off-by: Ben Ye <benye@amazon.com> * add compression scheme const to remote index cache Signed-off-by: Ben Ye <benye@amazon.com> * address required comments Signed-off-by: Ben Ye <benye@amazon.com> * fix compression scheme name Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: upgrading logs for failed uploads to error (#6427) * FIX: upgrading log for failed upload to error Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * docs: added changelog entry Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> --------- Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * fix postings test Signed-off-by: Ben Ye <benye@amazon.com> * Add aiven as adopter... more soon! Signed-off-by: Jonah Kowall <jkowall@kowall.net> * First draft of the Aiven adoption story Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add aiven as adopter... more soon! (#6430) Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Report gRPC connnection errors to the caller (#6428) By default `grpc.DialContext()` is non-blocking so any connection issue will not be surfaced to the user. This change makes it blocking and configures the gRPC dialer to report the underlying error if any happens. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * chore: remove duplicated `gopkg.in/fsnotify.v1` dep (#6432) * chore: remove duplicated `gopkg.in/fsnotify.v1` dep `github.com/fsnotify/fsnotify` and `gopkg.in/fsnotify.v1` are the same dependency. We can keep `github.com/fsnotify/fsnotify` and remove `gopkg.in/fsnotify.v1`. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * docs: add changelog Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> --------- Signed-off-by: Eng Zer Jun <engzerjun@gmail.com> * Expose estimated chunk and series size as configurable options (#6426) * expose estimated chunk and series size as configurable options Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Receive: make tsdb stats limit configurable (#6437) * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * Receive: make tsdb stats limit configurable Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> * *: wire new Engine/Explain fields in query-frontend (#6433) - Pass Engine/Explain fields in query-frontend codecs - Add Engine field to QFE cache key - Add e2e tests for all cases Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index cache: Cache expanded postings (#6420) * cache expanded postings in index cache Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * fix Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * rebase main and added compression name to key Signed-off-by: Ben Ye <benye@amazon.com> * update key Signed-off-by: Ben Ye <benye@amazon.com> * add e2e test for memcached Signed-off-by: Ben Ye <benye@amazon.com> * fix cache config Signed-off-by: Ben Ye <benye@amazon.com> * address review comments Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * add approximate series size to index stats (#6425) Signed-off-by: Ben Ye <benye@amazon.com> * index stats: fix chunk size calculation (#6424) Signed-off-by: Ben Ye <benye@amazon.com> * Remove some unused Cortex vendored code and metrics (#6440) * Avoid direct cortex vendor dependency in promclient (#6443) * use own explanation struct Signed-off-by: Ben Ye <benye@amazon.com> * omit empty Signed-off-by: Ben Ye <benye@amazon.com> * fix e2e test Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * query.md: Added grpc_{client,server}_handled_total to metrics list. (#6351) Fixes: #6280 Signed-off-by: Jonathan Davies <jpds@protonmail.com> * Updates busybox SHA (#6448) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * store: read postings directly into delta encoded format (#6442) Instead of allocating bytes for raw postings, let's read them directly into diff varint format to save memory. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * compact: Add Index Stats to block metadata (#6441) * compactor set index stats on block Signed-off-by: Ben Ye <benye@amazon.com> * comment Signed-off-by: Ben Ye <benye@amazon.com> * change field Signed-off-by: Ben Ye <benye@amazon.com> * update downsample Signed-off-by: Ben Ye <benye@amazon.com> * add index stats into compaction Signed-off-by: Ben Ye <benye@amazon.com> * fix tests Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * estimate block chunk and series size from metadata (#6449) Signed-off-by: Ben Ye <benye@amazon.com> * Adding graphic for blog showing architecture Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing mermaid graph again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Remove image and fix mermaid for other flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing prod flowchart title Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing md table again Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding clarification on cost breakdowns Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Store: fix crash on empty regex matcher Signed-off-by: Michael Hoffmann <michael.hoffmann@aiven.io> * Some corrections on text and links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing links Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing currencies Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing diagams Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing images Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Website: Fix blog images width (#6450) * Set blog post images width to 100% Blog Post images width isn't set and hence the images doesn't get fit to the page. Setting width to 100% make sure images fit to the web page and screen size. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Remove width constraint from blog images Signed-off-by: Pradyumna Krishna <git@onpy.in> --------- Signed-off-by: Pradyumna Krishna <git@onpy.in> * Cache calculated mint and maxt for each remote engine (#6458) The distributed optimizer in the Thanos engine calls MinT() and LabelSets multiple times on each remote engine. These calculations can get expensive when an engine covers a large amount of TSDBs. This commit introduces caching of the calculated values for a remote engine's mint, maxt and label sets. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cache empty expanded postings (#6464) Signed-off-by: Ben Ye <benye@amazon.com> * mixin/receive: add limits alerting (#6466) * add some thanos receive limits alerts Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * update changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix and generate examples Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * quick fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix test Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix invalid changelog Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * Update docs/blog/2023-06-08-thanos-at-aiven.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add retention for M3 Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Add image versus mermaid diagram. Couldn't comment it out unfortunately Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Adding text around link to fix linter Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> * add alert for tenant reaching head series limit (#6467) Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * go.mod: bump rest of otel libs (#6447) * go.mod: bump rest of otel libs Bump these two OTEL libs to be like the rest of the OTEL libs. We had a strange issue where `service.name` was not set even though it was in environment variables. Bumping these libs fixes the problem. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * otlp: only set service.name if provided Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * e2e: add test case Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * store: disable pooling for postings benchmarks (#6473) Pooling hides the real allocations in postings benchmarks because the allocation only happens once really. In reality this is different when multiple readers are created at once. Disable pooling in benchmarks to understand what is happening. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Fixing date and author for Aiven blog (#6476) * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> * make docs and fixing date Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Fixing authors Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Check context when expanding postings 2nd attempt (#6471) * check context when expanding postings Signed-off-by: Ben Ye <benye@amazon.com> * import Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * [FEAT] Add thanos ruler blog post (#6477) Signed-off-by: Hélia Barroso <helia.barroso@farfetch.com> * blog: Small fixes to Aiven article (#6481) * Correct some links and typos in 2023-06-08-thanos-at-aiven.md The linkified URLs in the article were including the closing bracket, breaking the links. Make them proper Markdown links. Also fixed a few other typos I spotted. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Update 2023-06-08-thanos-at-aiven.md Clarify a couple of things in the final paragraph. Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> --------- Signed-off-by: Hugh Cole-Baker <256988+sigmaris@users.noreply.github.com> * Rename series and postings fetch duration metrics (#6479) * fix wrong metric names for series and postings fetch duration Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * note breaking change Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Update to prometheus/common v0.44.0 (#6483) Signed-off-by: Dan Bason <colonel.dan@gmail.com> * go.mod: update rueidis client (#6485) The latest improvements have great perf improvements. Let's update it. Thank you to [rueian](https://github.com/rueian) and all of the contributors to that awesome Redis client! Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Introduce tenancy package (#6482) This commit introduces a tenancy package in preparation for introduction of tenancy in the query path. Tenancy related code which can be common between components are moved out of the receive component into the tenancy package. Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com> * Ignore nginx.com links in mdox (#6489) Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * blog: Update Aiven blog post (#6487) Wording changes and better formatting for the tables. Signed-off-by: Simon Brüggen <simon.bruggen@aiven.io> * store: optimized snappy streamed reading (#6475) Optimize snappy streamed reading by traversing through the byte slice in advance to determine how big of a buffer we will need. I have opted to rewrite snappy streamed format decoding because it is a straightforward format. Complex parts were deferred to klauspost/compress. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Remove Exists call in meta fetcher (#6474) * Remove Exists call in meta fetcher The meta fetcher lists the bucket by iterating through top level paths from the entire keyspace, and then for each key calls Exists on the meta.json file. This leads to an N+1 amplification against object storage, manifesting as a very high increase in costs and occasional throttling by cloud providers. This commit changes the iteration logic to recursively list all files instead, which can be done with far fewer API calls to object storage. Meta files are then identified from the keys returned by the list operation instead of making one API request from each file. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Revert e2e test change Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * e2e/store: use now instead of time.Now() each time (#6493) Consistently use the same timestamp instead of the current time each time during a test. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * [FIX] Documentation issue (#6494) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com> * Updates busybox SHA (#6497) Signed-off-by: GitHub <noreply@github.com> Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com> * chore: pkg imported more than once (#6499) * chore: pkg imported more than once Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * fix: remove pkg alias Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> --------- Signed-off-by: guoguangwu <guoguangwu@magic-shield.com> * *: Remove unnecessary configuration reload from `ContentPathReloader` and improve its tests (#6496) * Fix and improve PathContentReloader tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Run tests in parallel Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix table tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> * add context check when decoding cached postings (#6506) Signed-off-by: Ben Ye <benye@amazon.com> * *: Change the engine behind `ContentPathReloader` to be completely independent of any filesystem concept. (#6503) * Add fallback config reload for symlinks Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve pooling config reload Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make tests more reliable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve tests again Assert on runutil.Repeat output Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Pass the debounce/reload time to each test Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add comments and ensure interfaces are implemented Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make the limit configuration reload time customizable Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * goimports file Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix lint warning Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Improve log for polling engine Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Make linter happy Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Extract symlink identification to a separate function Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix links to go-grpc-middleware after v2 merge into main Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Add period to make linter happy. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog entry Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Replace the fsnotify based engine from `PathContentReloader` with the polling one Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove check for empty filePath Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Unexpose pollingEngine.Start Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix pollingEngine doc-comment Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Log file path when config is reloaded Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * receive: add float histogram support (#6323) * Added receive float histogram support Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Fixed imports Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Added comments for fns copied from Prometheus Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Improved comment Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unnecessary if Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fixed native histogram proto conversion in remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Removed unused histogram conversion from remote engine Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Fix and renaming in native_histograms_test.go Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * Trigger Build Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> --------- Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> * go.mod: bump e2e framework version (#6516) Bump e2e framework version and add a note about E2E_DOCKER_CPUS. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Add changelog for #6474 (#6518) Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * cortex/redisclient: use rueidis client (#6520) * cortex/redisclient: use rueidis client Use the same rueidis client in query-frontend. Solves https://github.com/thanos-io/thanos/issues/6094. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cortex/cache: gofumpt + fix errors.Errorf Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * cacheutil/docs: clean up more old stuff Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * index header: Remove memWriter from fileWriter (#6509) * index header: remove memWriter from fileWriter Signed-off-by: Ben Ye <benye@amazon.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> * refactor Signed-off-by: Ben Ye <benye@amazon.com> * fix test Signed-off-by: Ben Ye <benye@amazon.com> * update comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> * Wrap object store Prometheus registry (#6152) * Wrap object store Prometheus registery In preparation for the work being done at https://github.com/thanos-io/objstore/pull/26. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update objstore to the latest version This version removes the `thanos_` prefix from metrics, which is the reason for wrapping the objstore's metrics registry in the first place. Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix modules Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Wrap metrics registerer for objstore bucket Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Remove prefix from Thanos Store metrics Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix goimports Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Put back upgraded objstore dep Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Rerun CI Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Move to more recent ref of thanos/objstore Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Ignore OBS in objstore tests Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix linting error on main test after objstore upgrade Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Skip OCS objstore test in circle ci Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix echo in makefile Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Fix typo Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * e2e/store: try to fix Series() limit test again (#6522) I finally managed to reproduce this failure locally with http…

pull-request-size bot added the size/L label Apr 20, 2023

GiedriusS force-pushed the add_streamed_snappy_postings_decoding branch from abb02e5 to 24de10b Compare April 20, 2023 13:06

GiedriusS force-pushed the add_streamed_snappy_postings_decoding branch from 24de10b to 51a984d Compare April 20, 2023 13:27

pjarmalavicius previously approved these changes Apr 20, 2023

View reviewed changes

yeya24 reviewed Apr 20, 2023

View reviewed changes

GiedriusS commented Apr 25, 2023

View reviewed changes

pkg/store/postings_codec.go Outdated Show resolved Hide resolved

matej-g reviewed Apr 25, 2023

View reviewed changes

pkg/store/postings_codec.go Outdated Show resolved Hide resolved

pkg/store/bucket.go Outdated Show resolved Hide resolved

pkg/store/postings_codec.go Show resolved Hide resolved

pkg/store/postings_codec.go Outdated Show resolved Hide resolved

store: clean up postings code

05c3a43

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS dismissed pjarmalavicius’s stale review via 05c3a43 April 25, 2023 10:31

store: fix estimation

19d8554

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

fpetkovski reviewed Apr 25, 2023

View reviewed changes

GiedriusS added 3 commits April 25, 2023 15:36

store: use buffer.Bytes()

1ce500a

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

store/postings_codec: reuse extgrpc compressors/decompressors

4a037bd

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

CHANGELOG: add item

f9c99b7

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

yeya24 previously approved these changes Apr 25, 2023

View reviewed changes

fpetkovski previously approved these changes Apr 26, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into add_streamed_snappy_p…

57b26ee

…ostings_decoding Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS dismissed stale reviews from fpetkovski and yeya24 via 57b26ee April 26, 2023 08:22

CHANGELOG: clean up whitespace

25bcdc8

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS requested a review from fpetkovski April 26, 2023 09:12

fpetkovski approved these changes Apr 26, 2023

View reviewed changes

GiedriusS merged commit ba82514 into main Apr 26, 2023

GiedriusS deleted the add_streamed_snappy_postings_decoding branch April 28, 2023 14:49

This was referenced May 31, 2023

Store Gateway: Unexpected postings length #6402

Closed

Index Cache: Change cache key for postings #6405

Merged

yeya24 mentioned this pull request Jun 9, 2023

Streaming postings decoding leads to OOMs #6434

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

store: add streamed snappy encoding for postings list #6303

store: add streamed snappy encoding for postings list #6303

GiedriusS commented Apr 20, 2023 •

edited

Loading

pjarmalavicius left a comment

GiedriusS commented Apr 20, 2023 •

edited

Loading

yeya24 left a comment

GiedriusS commented Apr 25, 2023 •

edited

Loading

matej-g left a comment

fpetkovski left a comment

fpetkovski Apr 25, 2023

GiedriusS Apr 25, 2023

fpetkovski Apr 25, 2023

fpetkovski Apr 25, 2023

GiedriusS Apr 25, 2023

fpetkovski Apr 25, 2023

GiedriusS Apr 25, 2023

yeya24 commented Apr 25, 2023

yeya24 left a comment

GiedriusS commented Apr 25, 2023 •

edited

Loading

fpetkovski left a comment

fpetkovski left a comment


		uvarintEncodeBuf := make([]byte, binary.MaxVarintLen64)

		pooledSW := snappyWriterPool.Get()

store: add streamed snappy encoding for postings list #6303

store: add streamed snappy encoding for postings list #6303

Conversation

GiedriusS commented Apr 20, 2023 • edited Loading

pjarmalavicius left a comment

Choose a reason for hiding this comment

GiedriusS commented Apr 20, 2023 • edited Loading

yeya24 left a comment

Choose a reason for hiding this comment

GiedriusS commented Apr 25, 2023 • edited Loading

matej-g left a comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

fpetkovski Apr 25, 2023

Choose a reason for hiding this comment

GiedriusS Apr 25, 2023

Choose a reason for hiding this comment

fpetkovski Apr 25, 2023

Choose a reason for hiding this comment

fpetkovski Apr 25, 2023

Choose a reason for hiding this comment

GiedriusS Apr 25, 2023

Choose a reason for hiding this comment

fpetkovski Apr 25, 2023

Choose a reason for hiding this comment

GiedriusS Apr 25, 2023

Choose a reason for hiding this comment

yeya24 commented Apr 25, 2023

yeya24 left a comment

Choose a reason for hiding this comment

GiedriusS commented Apr 25, 2023 • edited Loading

fpetkovski left a comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

GiedriusS commented Apr 20, 2023 •

edited

Loading

GiedriusS commented Apr 20, 2023 •

edited

Loading

GiedriusS commented Apr 25, 2023 •

edited

Loading

GiedriusS commented Apr 25, 2023 •

edited

Loading