Receive: Tenants' external labels proposal #5720

haanhvu · 2022-09-25T19:15:57Z

Signed-off-by: haanhvu haanh6594@gmail.com

Proposal for #5434

haanhvu · 2022-09-25T19:17:02Z

@fpetkovski @saswatamcode This probably needs more details. Let's discuss this tomorrow.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

saswatamcode

Thanks for awesome work!

Some comments!

docs/proposals-accepted/202209-receive-tenant-external-labels.md

haanhvu · 2022-10-01T07:44:38Z

@fpetkovski I'll push something here at the end of the week. So don't review this yet if you don't have much time.

haanhvu · 2022-10-07T16:23:04Z

@fpetkovski @saswatamcode @yeya24 I made some changes. Please help review.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

fpetkovski

I like this proposal, simple and straight to the point. We need to polish out some typos, but other than that looks quite good already

docs/proposals-accepted/202209-receive-tenant-external-labels.md

saswatamcode

Thanks! Great work! 🙂

Some small nits and typos from above.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

yeya24

I am wondering if the following dynamic external label is part of the scope or not.
User can specify the external label names they want to extract as external labels to blocks. Upon receiving series from remote write we automatically group new blocks based on the external label value.

This sounds kind of hard but would be a super useful feature not only in Thanos but also in Cortex.

docs/proposals-accepted/202209-receive-tenant-external-labels.md

fpetkovski · 2022-11-14T08:14:02Z

We can start with statically defined labels first. Once that is in place, it would be interesting to explore extracting external labels from timeseries data.

yeya24 · 2022-11-20T20:02:24Z

Hey @haanhvu, could you please address the comments and we can merge the proposal?

haanhvu · 2022-11-20T20:07:05Z

Hey @haanhvu, could you please address the comments and we can merge the proposal?

@yeya24 Yes. I'll also add your idea as a follow-up^^

yeya24 · 2022-12-19T06:16:05Z

Ping @haanhvu. It is better to resolve the comments and merge the proposal before merging the implementation pr.

haanhvu · 2022-12-19T08:58:41Z

@yeya24 Yeah sorry for the delay I'm taking some days off. I'll update this proposal this week don't worry^^

Signed-off-by: haanhvu <haanh6594@gmail.com>

haanhvu · 2022-12-22T16:23:41Z

@fpetkovski @saswatamcode @yeya24 I updated the proposal. Besides the suggested wording changes, I also made some other changes based on what we've learned and discussed in the past months.

Please help review!

Signed-off-by: haanhvu <haanh6594@gmail.com>

yeya24

Thanks @haanhvu. This looks good to me. Anyone wants to take another review before I merge it?

fpetkovski

Looks good to me too 🚀

We can fix docs on main.

yeya24

Thanks. Let's fix doc on main.

* Receive external labels proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Restructure and edit proposal's content Signed-off-by: haanhvu <haanh6594@gmail.com> * Update proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Fix doc error Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com>

* Update Thanos engine to latest version (thanos-io#6069) This commit updates the Thanos PromQL engine to the latest version. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Receive: Tenants' external labels proposal (thanos-io#5720) * Receive external labels proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Restructure and edit proposal's content Signed-off-by: haanhvu <haanh6594@gmail.com> * Update proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Fix doc error Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com> * fixing doc CI (thanos-io#6072) Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Ben Ye <benye@amazon.com> * Fix stores filtering resets on reload (thanos-io#6063) * Fix stores filtering resets on reload `g0.store_matches` parameter appears in the url but doesn't applies in the frontend. Looks like it has been done on purpose and by removing a small piece of code fixes this issue. variable named `debugMode` is used for the store filtering checkbox which is an unappropriate name. Using `enableStoreFiltering` variable to represent the state of checkbox. Signed-off-by: Pradyumna Krishna <git@onpy.in> * Regenerate bindata.go Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Pradyumna Krishna <git@onpy.in> * Store: Make initial sync more robust Added re-try mechanism for store inital sync, where if the initial sync fails, it tries to do the initial sync again for given timeout duration. Signed-off-by: Kartik-Garg <kartik.garg@infracloud.io> * Recover from panics in Series calls (thanos-io#6077) * Recover from panics in Series calls This commit adds panic recovery for Series calls in all Store servers. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * Apply error suggestion Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> * query: reuse our own gate (thanos-io#6079) Do not call promgate directly but rather use our own wrapper that does everything we want - duration histogram, current in-flight calls, total calls. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * Store: Support disable cache index header file. (thanos-io#5773) * Store: Support disable cache index header file. Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Store: add a seprate flag to disable caching index header file Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Tools: add cleanup API for bucket web Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * resolve conversation Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * resolve confilcts Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * change the flag to `--cache-index-header` Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * Wrap mem writer in file writer Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * update CHANGELOG Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * update CHANGELOG Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> * fix bug Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> --------- Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> Co-authored-by: wanjunlei <wanjunlei@yujnify.com> * CVE: Fix Receiver malicious tenant (thanos-io#5969) If running as root or with enough privileges, receiver can create a directory outside of the configured TenantHeader. This commit fixes it up by sanitizing the user input and explicity not allowing such behavior. Signed-off-by: Daniel Mellado <dmellado@redhat.com> * Add adopter Grupo MasMovil (thanos-io#6084) Signed-off-by: Pablo Moncada Isla <pablo.moncada@masmovil.com> * fix typo (thanos-io#6087) Signed-off-by: cyip <cyip@jackhenry.com> Co-authored-by: cyip <cyip@jackhenry.com> * optimize selector to string (thanos-io#6076) Signed-off-by: Kama Huang <kamatogo13@gmail.com> * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <alanprot@gmail.com> * Adding tests Signed-off-by: Alan Protasio <alanprot@gmail.com> * reverting the change on get series Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * Cut 0.30.2 (thanos-io#6081) * tracing: fixed panic because of nil sampler (thanos-io#6066) * fixed panic because of nil sampler Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * added CHANGELOG entry Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> * bump version to 0.30.2 Signed-off-by: Ben Ye <benye@amazon.com> * Updates busybox SHA (thanos-io#6046) Signed-off-by: GitHub <noreply@github.com> Signed-off-by: GitHub <noreply@github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> * Use `e2edb.NewMinio` to disable SSE-S3 in e2e tests (thanos-io#6055) * Use e2edb.NewMinio to disable SSE Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Use temp fork for TLS Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix broken rules api fanout test Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix broken query compatibility test Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Remove fork Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> --------- Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * cherry pick store gateway fix to release 0.30 (thanos-io#6089) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock (thanos-io#6086) * Fix: Failure to close BlockSeriesClient cause store-gateway deadlock Signed-off-by: Alan Protasio <alanprot@gmail.com> * Adding tests Signed-off-by: Alan Protasio <alanprot@gmail.com> * reverting the change on get series Signed-off-by: Alan Protasio <alanprot@gmail.com> * fix lint Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> * update changelog Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> * fix changelog entries Signed-off-by: Ben Ye <benye@amazon.com> * docs: improving the description for tsdb.retention on the receiver Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> * Receiver: Use `intern` package when reallocating label strings (thanos-io#5926) * Cleanup go mod Signed-off-by: Matej Gera <matejgera@gmail.com> * Use string interning for labels realloc method Signed-off-by: Matej Gera <matejgera@gmail.com> * Enhance label realloc benchmarks Signed-off-by: Matej Gera <matejgera@gmail.com> * Make interning optional; put behind hiddend flag Signed-off-by: Matej Gera <matej.gera@coralogix.com> * Update CHANGELOG Signed-off-by: Matej Gera <matej.gera@coralogix.com> * Address feedback - Fix wrong condition - Adjust benchmarks Signed-off-by: Matej Gera <matej.gera@coralogix.com> --------- Signed-off-by: Matej Gera <matejgera@gmail.com> Signed-off-by: Matej Gera <matej.gera@coralogix.com> Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> * Updaing README with drawing fixes and minor wording clarification (thanos-io#6078) * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <jkowall@kowall.net> * New drawing and wording for Thanos other deployment models Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Added comments to README.md and updated the quick-tutorial.md with the same diagram updates and text to match Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Ran make docs Signed-off-by: Jonah Kowall <jkowall@kowall.net> --------- Signed-off-by: Jonah Kowall <jkowall@kowall.net> * Compact: Remove spam of replica label removed log (thanos-io#6088) * Remove spam of replica label removed log Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reduce amount of removed replica label instead of removing it Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Reformat code Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Store: Don't error when no stores are matched (thanos-io#6082) It's normal and not an error if a query does not match due to no downstream stores. This is common when querying with external labels and tiered query servers. This bug was introduced in thanos-io#5296 Fixes: thanos-io#5862 Signed-off-by: SuperQ <superq@gmail.com> * docs: Fix must have Ruler alerts definition (thanos-io#6058) * Fix must have Ruler alerts definition ThanosRuler missing rule intervals metric used the wrong comparator sign, confusing users trying to create the rule. Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> * Update docs/components/rule.md Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> --------- Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> * Fix conflicts Signed-off-by: haanhvu <haanh6594@gmail.com> * Specify overwriting behavior in flag and add validation Signed-off-by: haanhvu <haanh6594@gmail.com> * Add log and doc Signed-off-by: haanhvu <haanh6594@gmail.com> * Mixins(Rule): Fix query for long rule evaluations (thanos-io#6103) * mixin(Rule): Fix query for long rule evaluations Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> * Update changelog Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Pradyumna Krishna <git@onpy.in> Signed-off-by: Kartik-Garg <kartik.garg@infracloud.io> Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: wanjunlei <wanjunlei@kubesphere.io> Signed-off-by: Daniel Mellado <dmellado@redhat.com> Signed-off-by: Pablo Moncada Isla <pablo.moncada@masmovil.com> Signed-off-by: cyip <cyip@jackhenry.com> Signed-off-by: Kama Huang <kamatogo13@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com> Signed-off-by: Victor Fernandes <victorhbfernandes@gmail.com> Signed-off-by: Matej Gera <matejgera@gmail.com> Signed-off-by: Matej Gera <matej.gera@coralogix.com> Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Signed-off-by: Jonah Kowall <jkowall@kowall.net> Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: Maxim Muzafarov <m.muzafarov@gmail.com> Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com> Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: Ha Anh Vu <75315486+haanhvu@users.noreply.github.com> Co-authored-by: Ben Ye <benye@amazon.com> Co-authored-by: Pradyumna Krishna <git@onpy.in> Co-authored-by: Kartik-Garg <kartik.garg@infracloud.io> Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Co-authored-by: wanjunlei <53003665+wanjunlei@users.noreply.github.com> Co-authored-by: wanjunlei <wanjunlei@yujnify.com> Co-authored-by: Daniel Mellado <1313475+danielmellado@users.noreply.github.com> Co-authored-by: Pablo Moncada <pmoncadaisla@gmail.com> Co-authored-by: Chantel Yip <52993239+sshantel@users.noreply.github.com> Co-authored-by: cyip <cyip@jackhenry.com> Co-authored-by: Kama Huang <121007071+kama910@users.noreply.github.com> Co-authored-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: yeya24 <yeya24@users.noreply.github.com> Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com> Co-authored-by: Victor Fernandes <victorhbfernandes@gmail.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> Co-authored-by: Jonah Kowall <jkowall@kowall.net> Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com> Co-authored-by: Ben Kochie <superq@gmail.com> Co-authored-by: Maxim Muzafarov <m.muzafarov@gmail.com> Co-authored-by: haanhvu <haanh6594@gmail.com>

* Receive external labels proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Restructure and edit proposal's content Signed-off-by: haanhvu <haanh6594@gmail.com> * Update proposal Signed-off-by: haanhvu <haanh6594@gmail.com> * Fix doc error Signed-off-by: haanhvu <haanh6594@gmail.com> Signed-off-by: haanhvu <haanh6594@gmail.com>

pull-request-size bot added the size/M label Sep 25, 2022

yeya24 reviewed Sep 25, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

saswatamcode reviewed Sep 26, 2022

View reviewed changes

haanhvu force-pushed the proposal branch from fded92b to 9badd9b Compare October 7, 2022 16:20

haanhvu mentioned this pull request Oct 10, 2022

Receive: Allow specifying tenant-specific external labels in RouterIngestor #5777

Merged

2 tasks

yeya24 reviewed Oct 10, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

fpetkovski reviewed Oct 11, 2022

View reviewed changes

saswatamcode reviewed Oct 11, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

haanhvu commented Oct 19, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

yeya24 reviewed Nov 4, 2022

View reviewed changes

docs/proposals-accepted/202209-receive-tenant-external-labels.md Outdated Show resolved Hide resolved

haanhvu added 3 commits December 22, 2022 23:18

Receive external labels proposal

40ed4b5

Signed-off-by: haanhvu <haanh6594@gmail.com>

Restructure and edit proposal's content

08cc3a2

Signed-off-by: haanhvu <haanh6594@gmail.com>

Update proposal

54e6af0

Signed-off-by: haanhvu <haanh6594@gmail.com>

haanhvu force-pushed the proposal branch from f5ec96c to 54e6af0 Compare December 22, 2022 16:18

haanhvu changed the title ~~[WIP] Receive external labels proposal~~ Receive: Tenants' external labels proposal Dec 22, 2022

Fix doc error

f1b1909

Signed-off-by: haanhvu <haanh6594@gmail.com>

yeya24 approved these changes Jan 15, 2023

View reviewed changes

matej-g approved these changes Jan 16, 2023

View reviewed changes

fpetkovski approved these changes Jan 16, 2023

View reviewed changes

yeya24 approved these changes Jan 24, 2023

View reviewed changes

yeya24 merged commit 5553244 into thanos-io:main Jan 24, 2023

yeya24 mentioned this pull request Jan 24, 2023

Fixing doc CI #6072

Merged

2 tasks

haanhvu mentioned this pull request May 25, 2023

Receive: Allow specifying tenant-specific external labels #5434

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Receive: Tenants' external labels proposal #5720

Receive: Tenants' external labels proposal #5720

haanhvu commented Sep 25, 2022 •

edited

Loading

haanhvu commented Sep 25, 2022

saswatamcode left a comment

haanhvu commented Oct 1, 2022

haanhvu commented Oct 7, 2022

fpetkovski left a comment

saswatamcode left a comment

yeya24 left a comment

fpetkovski commented Nov 14, 2022

yeya24 commented Nov 20, 2022

haanhvu commented Nov 20, 2022

yeya24 commented Dec 19, 2022

haanhvu commented Dec 19, 2022 •

edited

Loading

haanhvu commented Dec 22, 2022

yeya24 left a comment

fpetkovski left a comment

yeya24 left a comment

Receive: Tenants' external labels proposal #5720

Receive: Tenants' external labels proposal #5720

Conversation

haanhvu commented Sep 25, 2022 • edited Loading

haanhvu commented Sep 25, 2022

saswatamcode left a comment

Choose a reason for hiding this comment

haanhvu commented Oct 1, 2022

haanhvu commented Oct 7, 2022

fpetkovski left a comment

Choose a reason for hiding this comment

saswatamcode left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

fpetkovski commented Nov 14, 2022

yeya24 commented Nov 20, 2022

haanhvu commented Nov 20, 2022

yeya24 commented Dec 19, 2022

haanhvu commented Dec 19, 2022 • edited Loading

haanhvu commented Dec 22, 2022

yeya24 left a comment

Choose a reason for hiding this comment

fpetkovski left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

haanhvu commented Sep 25, 2022 •

edited

Loading

haanhvu commented Dec 19, 2022 •

edited

Loading