Implement federated metric metadata API #3686

yeya24 · 2020-12-30T22:45:34Z

Signed-off-by: Ben Ye yb532204897@gmail.com

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

This pr follows the awesome pr #3350 and comment #3345 (comment) to implement the Prometheus metric metadata API https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata.

Changes

Verification

Tested manually and E2E tests added.

bwplotka · 2021-02-02T09:36:58Z

I think this slipped through notification - this is greater, let's Review!

cc @lilic (in regards to Info API re-design), @kakkoyun @brancz @squat @GiedriusS

pkg/query/storeset.go

cmd/thanos/query.go

kakkoyun · 2021-02-12T10:27:43Z

@yeya24 is this ready for another round of review?

Also it worths having a look:

yeya24 · 2021-02-15T01:02:04Z

@kakkoyun Thanks! I have fixed that issue and this pr is ready for review.

juliusv · 2021-02-22T21:07:40Z

Reminder poke to the reviewers (this feature would be awesome to have) :)

onprem

Few suggestions but everything else LGTM. Let's wait for others to review though. Awesome work @yeya24!

cmd/thanos/query.go

pkg/api/query/v1.go

test/e2e/metadata_api_test.go

yeya24 · 2021-02-23T19:00:39Z

@prmsrswt Thanks for the detailed review. I have addressed your comments.

bwplotka

This is solid work, I think it's good for start TBH 🤗

I would consider merging this for 0.19. Objections @kakkoyun @squat @prmsrswt ?

bwplotka · 2021-02-24T18:51:15Z

pkg/metadata/metadatapb/rpc.proto

+
+message MetadataRequest {
+  string metric = 1;
+  int32 limit = 2;


What's the limit?

https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata It is used to limit the # of returned metadata.

pkg/metadata/metadatapb/rpc.proto

Signed-off-by: Ben Ye <yb532204897@gmail.com>

Signed-off-by: yeya24 <yb532204897@gmail.com>

kakkoyun

🥇

bwplotka · 2021-02-25T10:15:37Z

😱

* support federated metadata API Signed-off-by: Ben Ye <yb532204897@gmail.com> * update comments Signed-off-by: yeya24 <yb532204897@gmail.com> * use parseInt Signed-off-by: yeya24 <yb532204897@gmail.com> * address Prem's comments Signed-off-by: yeya24 <yb532204897@gmail.com> * update proto comment Signed-off-by: yeya24 <yb532204897@gmail.com> * add changelog Signed-off-by: yeya24 <yb532204897@gmail.com>

@squat

* CHANGELOG.md: add v0.16.0 link (thanos-io#3697) This small commit fixes the changelog entry for release v0.16.0 so that the section title includes a link to the actual release. Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * Delete deletion-mark.json at last when deleting a block (thanos-io#3661) * Delete deletion-mark.json at last when deleting a block Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed linter Signed-off-by: Marco Pracucci <marco@pracucci.com> * fixed broken links (thanos-io#3645) Signed-off-by: Namanl2001 <namanlakhwani@gmail.com> * chore(tutorials): fix some broken links in 2-lts (thanos-io#3702) Signed-off-by: Bradley <bwilsonhunt@gmail.com> * Fix race condition in BinaryReader.LookupSymbol() (thanos-io#3705) * Fix race condition in BinaryReader.LookupSymbol() Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed BinaryReader receiver and added CHANGELOG entry Signed-off-by: Marco Pracucci <marco@pracucci.com> * ui: make old bucket viewer UI work with vanilla blocks (thanos-io#3700) * ui: make old UI work with vanilla blocks Make the old bucket viewer UI work with vanilla Prometheus blocks by checking whether the `Thanos` part exists before formatting the HTML. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * CHANGELOG: update Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * ui/bucket: fix according to @squat's suggestions Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * ui: update bindata after latest changes Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Query-frontend CMD: use detailed text description Signed-off-by: dmaiocchi <sodar@riseup.net> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Upgrade Cortex and remove SwiftConfig (thanos-io#3708) Signed-off-by: Marco Pracucci <marco@pracucci.com> * chore(deps): upgrading hugo (v0.80.0) fixes thanos-io#3653 (thanos-io#3714) Signed-off-by: Bradley <bwilsonhunt@gmail.com> * store: Make more S3 http.Transport settings configurable. (thanos-io#3657) The http.Transport is not auto-tuning and one size does not seem to fit all cases. In order to respond to a query a store gateway might need to fetch a large (thousands) of postings, series and chunks. While the number of idle connections has been increased recently it can still be too low or expose bursty (opening, closing) behavior. Allow to tune most of the http.Transport parameters. I considered embedding the full Transport but there is already a Transport member. Signed-off-by: Holger Hans Peter Freyther <holger@freyther.de> * added definiton of chunk and referenced it in the docs (thanos-io#3629) * Update design.md Update store.md Update design.md Update storage.md Update troubleshooting.md Signed-off-by: Biswajit Ghosh <biswajitghosh98@iitkgp.ac.in> * Update design.md Update store.md Update design.md Update troubleshooting.md Update storage.md Signed-off-by: Biswajit Ghosh <biswajitghosh98@iitkgp.ac.in> * Upgraded to newest bingo. (thanos-io#3718) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Update help messages with a consistent use of capitals and periods. (thanos-io#3727) * Update help messages with a consistent use of capitals and periods. Signed-off-by: Matt Whitney <mattjwhitney@gmail.com> * Update documentation to reflect message changes. Signed-off-by: Matt Whitney <mattjwhitney@gmail.com> * Run `make docs` to correct the documentation rather than manual edits. Signed-off-by: Matt Whitney <mattjwhitney@gmail.com> * docs: update examples/dashboards/dashboards.md (thanos-io#3733) Signed-off-by: Mert Acikportali <mertacikportali@gmail.com> * pkg/errutil: correct the multierror file suffix (thanos-io#3736) This commit changes the file suffix from .go.go to simply .go. Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * Fixed website and added new step for release process (tmp). (thanos-io#3738) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * api: Added Proposal for Discovery Endpoint idea (InfoAPI) (thanos-io#3703) * docs/proposals/210701_endpoint_discovery.md: Add initial proposal Signed-off-by: Lili Cosic <cosiclili@gmail.com> * Update docs/proposals/210701_endpoint_discovery.md Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Introduce "block-meta-fetch-concurrency" flag for compact and store (thanos-io#3752) * Introduce "block-meta-fetch-concurrency" flag for compact and store Furthermore consolidate the fetcher concurrency by moving the fetcherConcurrency const to the fetcher code of the block package. Signed-off-by: Johannes Frey <johannes.frey@daimler.com> * Document flags Signed-off-by: Johannes Frey <johannes.frey@daimler.com> * Add `--query-range.request-downsampled` flag to Query Frontend (thanos-io#2641) (thanos-io#3723) * Add `--query-range.request-downsampled` flag to Query Frontend (thanos-io#2641) Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Pinned to newer busybox on quay. (thanos-io#3762) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix panic on concurrent index-header lazy reader usage and unload (thanos-io#3760) * Fix panic on concurrent index-header lazy reader usage and unload Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed Planner.Plan() description (thanos-io#3767) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed panic on concurrent index-header lazy load / unload (thanos-io#3759) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Pad compaction planner size check (thanos-io#3773) Pad the index size check in the compaction planner by 15% to avoid situations where the sum of index bytes bloats to be larger than the sum of the original index sizes. * thanos-io#3724 * thanos-io#3750 Signed-off-by: Ben Kochie <superq@gmail.com> * pkg/server/http: Allow passing http.ServeMux with a server option (thanos-io#3769) Signed-off-by: Matthias Loibl <mail@matthiasloibl.com> * Update Katacoda Official Link (thanos-io#3768) Signed-off-by: soniasingla <soniasingla.1812@gmail.com> * Nomitate kakkoyun for the v0.20.0 release (thanos-io#3778) Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix typo in query component documentation for store api link (thanos-io#3675) * Fix wrong store api link Signed-off-by: Junyoung, Sung <junyoung.sung@navercorp.com> * Fix store api link to absolute Signed-off-by: Junyoung, Sung <junyoung.sung@navercorp.com> * e2e: Resolved Flaky tests (thanos-io#3777) * added retry function Signed-off-by: Abhishek357 <abhisheksinghchauhan357@gmail.com> * fixed CI test Signed-off-by: Abhishek357 <abhisheksinghchauhan357@gmail.com> * restructured code Signed-off-by: Abhishek357 <abhisheksinghchauhan357@gmail.com> * Add an exempt label to ignore issue that has a PR Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * in memory cache for caching bucket (thanos-io#3579) * in memeory cache for caching bucket Signed-off-by: Sudharshann D <sudhar287@gmail.com> * addressing comments Signed-off-by: Sudharshann D <sudhar287@gmail.com> * Introduce "allow overlapping blocks" flag for Thanos receiver (thanos-io#3792) * add tsdb.allow-overlapping-blocks flag to receiver Signed-off-by: Max Chandler <max_chandler@apple.com> * make format Signed-off-by: Max Chandler <max_chandler@apple.com> * cleanup whitespace Signed-off-by: Max Chandler <max_chandler@apple.com> * cleanup whitespace Signed-off-by: Max Chandler <max_chandler@apple.com> * make docs Signed-off-by: Max Chandler <max_chandler@apple.com> * fix changelog conflict Signed-off-by: Max Chandler <max_chandler@apple.com> Co-authored-by: Matt Lawrence <m.law@apple.com> * Reduce memory allocations in bucketBlock.readChunkRange() (thanos-io#3796) Signed-off-by: Marco Pracucci <marco@pracucci.com> * update prometheus version Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * update cortex Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * add new property to LabelValuesRequest Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * set label matchers in LabelValueRequest Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * make docs Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * Allow to customise S3 SSE on a per-request basis (thanos-io#3783) * Allow to customise S3 SSE on a per-request basis Signed-off-by: Marco Pracucci <marco@pracucci.com> * Addressed review comments Signed-off-by: Marco Pracucci <marco@pracucci.com> * Refactoring: allow to pass BytesPool to store.NewBucketStore() (thanos-io#3801) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Allow to customise the partitioner used by the BucketStore (thanos-io#3802) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix button display when there is no panels (thanos-io#3694) Signed-off-by: Namanl2001 <namanlakhwani@gmail.com> * fix tests Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * Allow downstream projects to customise the Partitioner (thanos-io#3808) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Makefile: Fix to command to find React source files (thanos-io#3805) Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Merge release 0.18 (thanos-io#3809) * CHANGELOG.md: release v0.18.0 Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * VERSION,tutorials: bump versions Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * Revert "VERSION,tutorials: bump versions" This reverts commit 5a27c70. The "thanos:" prefix for the container images was accidentally removed. Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * VERSION,tutorials: bump version for all images Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * CHANGELOG: bump release date Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * pkg/rules/proxy: fix hotlooping when receiving client errors Currently, if we receive an error from the underlying client stream, we continue with trying to receive additional data. This causes a hotloop as we will receive the same error again. This fixes it by returning in the error case and adds a unit test for the proxy logic. Fixes thanos-io#3717 Signed-off-by: Sergiusz Urbaniak <sergiusz.urbaniak@gmail.com> * CHANGELOG.md: fix changelog to incorporate new fix (thanos-io#3737) This commit fixes the order of the changelog to properly reflect a recent change. Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * Fixed website and added new step for release process (tmp). (thanos-io#3738) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * CHANGELOG.md: update v0.18.0 release date Signed-off-by: Lucas Servén Marín <lserven@gmail.com> Co-authored-by: Sergiusz Urbaniak <sergiusz.urbaniak@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * reloader: try to fix test flakiness (thanos-io#3798) This is an attempt of fixing the `TestReloader_DirectoriesApply` test flakiness. I call this an attempt because I cannot reproduce it locally. However, I have noticed that during runs where this fails the logs look like this: ``` --- FAIL: TestReloader_DirectoriesApply (3.04s) reloader_test.go:256: Performing step number 0 reloader_test.go:256: Performing step number 1 reloader_test.go:256: Performing step number 2 reloader_test.go:256: Performing step number 3 reloader_test.go:256: Performing step number 4 reloader_test.go:256: Performing step number 6 reloader_test.go:343: reloader_test.go:343: exp: 5 got: 6 ``` It immediately jumps to another value. This gave me a hint and I think that this is happening because potentially `i` can be written to/read from by multiple goroutines. On very resource constrained systems like the CircleCI runners, it could just happen that the `if` doesn't do what it is supposed to. Try to fix this problem by protecting the whole HTTP handler with a mutex. Since this is only a test and not a performance critical path, I think this is a reasonable change to do. Add extra check for reload failures. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * addressing PR comments Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> * Block Viewer: Move overlapping blocks to separate rows (thanos-io#3729) * Block Viewer: overlapping blocks issue resolved Signed-off-by: Namanl2001 <namanlakhwani@gmail.com> * small changes and make assets Signed-off-by: Namanl2001 <namanlakhwani@gmail.com> * Add flag to set default step Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Add support to use default step in classic ui Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Add support to use default step in new ui Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Lint fixes Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * change ParseStep function to be consistent with ui Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Improve description of query.default-step flag Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Minor fixes Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Use default value when flag is undefined Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * minor fixes Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Minor fixes Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Update changelog Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Fix typo Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Receive: Improve handling of empty time series from clients (thanos-io#3815) * exit early if a request has no timeseries data Signed-off-by: Matt Lawrence <m.law@apple.com> Co-authored-by: Max Chandler <max_chandler@apple.com> * update changelog Signed-off-by: Matt Lawrence <m.law@apple.com> * update changelog Signed-off-by: Matt Lawrence <m.law@apple.com> * Move constant to right side of comparison Co-authored-by: Lucas Servén Marín <lserven@gmail.com> Signed-off-by: Matt Lawrence <m.law@apple.com> Co-authored-by: Max Chandler <max_chandler@apple.com> Co-authored-by: Lucas Servén Marín <lserven@gmail.com> * Reduced allocated memory by chunks reader in the store gateway at query time (thanos-io#3814) * Reduced allocated memory by chunks reader in the store gateway at query time Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed linter issues Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed linter (hopefully) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Renamed function Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated comment Signed-off-by: Marco Pracucci <marco@pracucci.com> * Updated code comment Signed-off-by: Marco Pracucci <marco@pracucci.com> * promclient: fix error's message (thanos-io#3824) Use the provided `method` in the error messages. I got scared reading the logs that the requests are still being sent using GET instead of POST which I had specified. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * pkg/ui/react-app: update dependencies (thanos-io#3818) Fix lodash security issue. Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Add objstore.List() recursive support (thanos-io#3823) * Add objstore.List() recursive support Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed CachingBucket Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix Cortex compilation issue Signed-off-by: Marco Pracucci <marco@pracucci.com> * Tidy go.mod/sum Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix linter Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed TestMetricBucket_Close Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Cortex Signed-off-by: Marco Pracucci <marco@pracucci.com> * Inlined struct initialisation Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgrade Cortex again Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgrade Cortex (thanos-io#3828) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Truncated S3 "get object" response is reported as error (thanos-io#3795) * Add test to ensure S3 truncated response is an error Signed-off-by: Marco Pracucci <marco@pracucci.com> * Added missing copyright Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Minio Signed-off-by: Marco Pracucci <marco@pracucci.com> * Upgraded Minio again Signed-off-by: Marco Pracucci <marco@pracucci.com> * Implement federated metric metadata API (thanos-io#3686) * support federated metadata API Signed-off-by: Ben Ye <yb532204897@gmail.com> * update comments Signed-off-by: yeya24 <yb532204897@gmail.com> * use parseInt Signed-off-by: yeya24 <yb532204897@gmail.com> * address Prem's comments Signed-off-by: yeya24 <yb532204897@gmail.com> * update proto comment Signed-off-by: yeya24 <yb532204897@gmail.com> * add changelog Signed-off-by: yeya24 <yb532204897@gmail.com> * Fixed TestBucketStore_ManyParts_e2e (thanos-io#3841) https://app.circleci.com/pipelines/github/thanos-io/thanos/5120/workflows/efd2a21d-13b7-4035-99e3-cb1af8023694/jobs/13809 This fail is only visible for anyone from Thanos Team proposing PR, due to extra tests against bucket providers. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Reinforcing errutil and fixed ugly bug which caused empty []error to te treated as error, (thanos-io#3836) Previous multi-error implementation could cause very ugly bug of returnig empty multi-error that should be treated as success not error by API, but if .Err() is not invoked it will be used as non nil error. Once we merge this, we can do cleaner solution that slighly change nesting behaviour: thanos-io#3833 There were 9 places where we had this bug in handler due to MultiError lib allowing to do so. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Export metadata fetcher metrics (thanos-io#3660) Signed-off-by: Marco Pracucci <marco@pracucci.com> * store: Cleaned up API for test/benchmark purposes. (thanos-io#3650) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Allow Cortex to fully reuse FetcherMetrics (thanos-io#3842) Signed-off-by: Marco Pracucci <marco@pracucci.com> * downsample: ensure consistent order (thanos-io#3843) Ensure that we have a consistent order of blocks that we are going to downsample. `range` over maps doesn't enforce any particular order on purpose. This is needed for https://github.com/thanos-io/thanos/pull/3031/files. ATM in that PR before downsampling we delete all directories which do not match blocks ULIDs in the remote object storage. Ideally, we should only keep around the files of a block which we are about to downsample. It is impossible to do that properly ATM if during another iteration we'd start from a different block. Thus, let's have a consistent order. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * [Fix] Replace unsupported %v macro in logs (thanos-io#3847) Signed-off-by: Stéphane Brillant <stephane.brillant@jive.com> * Update ui npm dependencies to latest versions (thanos-io#3813) * Update ui npm deps to latest Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix linting errors Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Change camelcase lint rule Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix CI configs and scripts to work with main branch (thanos-io#3848) Signed-off-by: Prem Saraswat <prmsrswt@gmail.com> * block: precalculate hashes if enabled and use them during compaction (downloading) (thanos-io#3031) * block: precalculate hashes if enabled and use them during compaction Added the possibility to ignore certain directories in objstore.{Download,DownloadDir}. Do not download files which have the same hash as in remote object storage. Wire up `--hash-func` so that writers could specify what hash function to use when uploading. There is no performance impact if no hash function has been explicitly specified. Clean up the removal of files logic in Thanos Compact to ensure we do not remove something that exists on disk already. Tested manually + new tests cover all of this more or less. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * block: expose GatherFileStats and use it Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Revert "block: expose GatherFileStats and use it" This reverts commit 259c70b. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * block: do not calc hash for dirs, add locks Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * docs/tools: update Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * shipper: pass s.hashFunc Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * compact: clean up comment Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * block: close with log on error Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * *: remove unused FNs Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * compact: add e2e test for new hash functionality Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * mixin: Upgrade jsonnet tooling (thanos-io#3855) * Upgrade jsonnet tooling Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix bingo version issue Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Fix unintended removal issue Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update reference of master to main in docs (thanos-io#3849) * Update reference of master to main in docs Signed-off-by: Prem Saraswat <prmsrswt@gmail.com> * Update image tag in tutorials/kubernetes-helm Signed-off-by: Prem Saraswat <prmsrswt@gmail.com> * Cut v0.19.0-rc.0 (thanos-io#3860) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed missed changelog comments. (thanos-io#3865) Missed from: thanos-io#3860 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * rename Metadata API to MetricMetadata API (thanos-io#3877) Signed-off-by: yeya24 <yb532204897@gmail.com> * Fix parseStep bug introduced in PR thanos-io#3740 (thanos-io#3887) (thanos-io#3889) * Fix bug introduced in PR thanos-io#3740 Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> * Minor change in test Signed-off-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> Co-authored-by: Hitanshu Mehta <44025541+hitanshu-mehta@users.noreply.github.com> * Tools: rewrite delete can delete series only if it matches all matchers (thanos-io#3886) (thanos-io#3890) * rewrite delete should delete a series only if it matches all matchers in a deletion request Signed-off-by: yeya24 <yb532204897@gmail.com> * add test case Signed-off-by: yeya24 <yb532204897@gmail.com> Co-authored-by: Ben Ye <yb532204897@gmail.com> * cmd/thanos/receive.go: Receive client infers TLS (thanos-io#3899) Currently, the thanos Receiver infers whether it should use TLS for the gRPC clients that forwards time series to other receivers based on whether the remote-write HTTP server uses TLS. This is not correct, as the HTTP server may use TLS without the gRPC server using TLS. This commit fixes the inference. Longer-term, this will be taken care of by the receive/router split. Signed-off-by: Lucas Servén Marín <lserven@gmail.com> * Cut v0.19.0-rc.1 (thanos-io#3900) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * tools: Fix partial and empty matchers in rewrite (thanos-io#3891) * Fix partial and empty matchers match Signed-off-by: yeya24 <yb532204897@gmail.com> * add testcase for non-equal matchers Signed-off-by: yeya24 <yb532204897@gmail.com> * v0.19.0 patch: Added receive benchmark; Fixed Receiver excessive mem usage introduced in 0.17 (thanos-io#3943) * Added receive benchmark, baseline. ``` goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK-12 22260 1550152 ns/op 1380340 B/op 6093 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors-12 6619 6430408 ns/op 4522487 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK-12 2695 17208794 ns/op 15072963 B/op 60441 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors-12 474 72533286 ns/op 46396932 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 270 137050518 ns/op 226595379 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 21 1616025443 ns/op 698724321 B/op 408 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Copy labels. ``` GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchmem -test.benchtime=30s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 25887 1537262 ns/op 1380023 B/op 6092 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4237 7547968 ns/op 4522583 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 2205 16513380 ns/op 15071092 B/op 60420 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 525 67278233 ns/op 46396645 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 285 148049189 ns/op 226596168 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 20 1731361499 ns/op 698722550 B/op 401 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addded bench., Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fix. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Improved API. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Changelog. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Lucas comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * compact: clean up directories thoroughly (thanos-io#3869) * compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in thanos-io#3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed comment. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> * Added benchmark, Moved minio-deps to fork without race fix we don't need. (thanos-io#3968) Fixes: thanos-io#3917 Long term fix: thanos-io#3967 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Cut v0.19.0-rc.2 (thanos-io#3969) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * pkg/rules: fix deduplication of equal alerts with different labels (thanos-io#3960) (thanos-io#3999) Currently, if an alerting rule having the same name with different severity labels is being returned from different replicas then they are being treated as separate alerts. Given the following alerts a1,a2 with severities s1,s2 returned from replicas r1,2: a1[s1,r1] a1[s2,r1] a1[s1,r2] a1[s2,r2] Then, currently, the algorithm deduplicates to: a1[s1] a1[s2] a1[s1] a1[s2] Instead of the intendet result: a1[s1] a1[s2] This fixes it by removing replica labels before sorting labels for deduplication. Signed-off-by: Sergiusz Urbaniak <sergiusz.urbaniak@gmail.com> # Conflicts: # CHANGELOG.md Co-authored-by: Sergiusz Urbaniak <sergiusz.urbaniak@gmail.com> * Cut v0.19.0 (thanos-io#3998) Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Fixed VERSION file. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Lucas Servén Marín <lserven@gmail.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Naman Lakhwani <namanlakhwani@gmail.com> Co-authored-by: Bradley <bwilsonhunt@gmail.com> Co-authored-by: Giedrius Statkevičius <giedriuswork@gmail.com> Co-authored-by: dmaiocchi <dmaiocchi@suse.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Ben Ye <yb532204897@gmail.com> Co-authored-by: Holger Freyther <holger@freyther.de> Co-authored-by: Biswajit Ghosh <34703680+Biswajitghosh98@users.noreply.github.com> Co-authored-by: Matt W <mattjwhitney@gmail.com> Co-authored-by: Mert Açıkportalı <mertacikportali@gmail.com> Co-authored-by: Lili Cosic <cosiclili@gmail.com> Co-authored-by: Johannes Frey <johannes.frey@daimler.com> Co-authored-by: Vladimir Kononov <krya-kryak@users.noreply.github.com> Co-authored-by: Ben Kochie <superq@gmail.com> Co-authored-by: Matthias Loibl <mail@matthiasloibl.com> Co-authored-by: Sonia Singla <soniasingla.1812@gmail.com> Co-authored-by: Kemal Akkoyun <kakkoyun@users.noreply.github.com> Co-authored-by: Junyoung, Sung <junyoung.sung@navercorp.com> Co-authored-by: Abhishek Singh Chauhan <abhisheksinghchauhan357@gmail.com> Co-authored-by: Sudhar287 <sudhar287@gmail.com> Co-authored-by: Max Chandler <59826076+Max-Chandler@users.noreply.github.com> Co-authored-by: Matt Lawrence <m.law@apple.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Hitanshu Mehta <44025541+hitanshu-mehta@users.noreply.github.com> Co-authored-by: Sergiusz Urbaniak <sergiusz.urbaniak@gmail.com> Co-authored-by: Hitanshu Mehta <hitanshu99amehta@gmail.com> Co-authored-by: Matt Lawrence <79317313+m-lawre@users.noreply.github.com> Co-authored-by: Max Chandler <max_chandler@apple.com> Co-authored-by: Simon Pasquier <spasquie@redhat.com> Co-authored-by: Stephane Brillant <stephane.brillant@logmein.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Prem Saraswat <prmsrswt@gmail.com>

yeya24 force-pushed the metadata-api branch 3 times, most recently from 0f791fa to 0530f22 Compare December 30, 2020 23:31

juliusv mentioned this pull request Feb 2, 2021

Implement metrics metadata API #3345

Closed

bwplotka requested review from onprem, kakkoyun and squat February 2, 2021 09:37

yeya24 commented Feb 3, 2021

View reviewed changes

pkg/query/storeset.go Outdated Show resolved Hide resolved

pkg/query/storeset.go Outdated Show resolved Hide resolved

cmd/thanos/query.go Outdated Show resolved Hide resolved

cmd/thanos/query.go Outdated Show resolved Hide resolved

onprem approved these changes Feb 23, 2021

View reviewed changes

cmd/thanos/query.go Outdated Show resolved Hide resolved

pkg/api/query/v1.go Outdated Show resolved Hide resolved

test/e2e/metadata_api_test.go Outdated Show resolved Hide resolved

bwplotka approved these changes Feb 24, 2021

View reviewed changes

yeya24 added 6 commits February 24, 2021 17:50

support federated metadata API

6ebaa1f

Signed-off-by: Ben Ye <yb532204897@gmail.com>

update comments

d18d9bb

Signed-off-by: yeya24 <yb532204897@gmail.com>

use parseInt

810929e

Signed-off-by: yeya24 <yb532204897@gmail.com>

address Prem's comments

1c10b72

Signed-off-by: yeya24 <yb532204897@gmail.com>

update proto comment

046a53c

Signed-off-by: yeya24 <yb532204897@gmail.com>

add changelog

3f7e42e

Signed-off-by: yeya24 <yb532204897@gmail.com>

yeya24 force-pushed the metadata-api branch from 46ecf4f to 3f7e42e Compare February 24, 2021 22:53

kakkoyun approved these changes Feb 25, 2021

View reviewed changes

kakkoyun merged commit 46d0106 into thanos-io:master Feb 25, 2021

yeya24 deleted the metadata-api branch February 25, 2021 15:21

squat mentioned this pull request Mar 2, 2021

Cut v0.19.0-rc.0 🎉 #3860

Merged

yeya24 mentioned this pull request Mar 4, 2021

Implement target metadata API #3870

Closed

This was referenced Jul 13, 2021

Add target metadata API #4438

Closed

query: Extend metadata service to include target metadata API #4449

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement federated metric metadata API #3686

Implement federated metric metadata API #3686

yeya24 commented Dec 30, 2020 •

edited

Loading

bwplotka commented Feb 2, 2021 •

edited

Loading

kakkoyun commented Feb 12, 2021

yeya24 commented Feb 15, 2021

juliusv commented Feb 22, 2021

onprem left a comment •

edited

Loading

yeya24 commented Feb 23, 2021

bwplotka left a comment

bwplotka Feb 24, 2021

yeya24 Feb 24, 2021

kakkoyun left a comment

bwplotka commented Feb 25, 2021

Implement federated metric metadata API #3686

Implement federated metric metadata API #3686

Conversation

yeya24 commented Dec 30, 2020 • edited Loading

Changes

Verification

bwplotka commented Feb 2, 2021 • edited Loading

kakkoyun commented Feb 12, 2021

yeya24 commented Feb 15, 2021

juliusv commented Feb 22, 2021

onprem left a comment • edited Loading

Choose a reason for hiding this comment

yeya24 commented Feb 23, 2021

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka Feb 24, 2021

Choose a reason for hiding this comment

yeya24 Feb 24, 2021

Choose a reason for hiding this comment

kakkoyun left a comment

Choose a reason for hiding this comment

bwplotka commented Feb 25, 2021

yeya24 commented Dec 30, 2020 •

edited

Loading

bwplotka commented Feb 2, 2021 •

edited

Loading

onprem left a comment •

edited

Loading