Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store/proxy: Deduplicate chunks on StoreAPI level. Recommend chunk sorting for StoreAPI + Optimized iter chunk dedup. (#2710) #2711

Merged
merged 1 commit into from
Jun 4, 2020

Conversation

bwplotka
Copy link
Member

@bwplotka bwplotka commented Jun 3, 2020

Cherry picked: #2710

  • Deduplicate chunk dups on proxy StoreAPI level. Recommend chunk sorting for StoreAPI.

Also: Merge same series together on proxy level instead select. This allows better dedup efficiency.

Partially fixes: #2303

Cases like overlapped data from store and sidecar and 1:1 duplicates are optimized as soon as it's possible.
This case was highly visible on GitLab repro data and exists in most of Thanos setup.

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

  • Optimized algorithm to combine series only on start.

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

  • Optimized chunk comparision for overlaps.

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

  • Optimized deduplication for deduplicated chunk on query level as well.

Never use proto .String() in fast path!

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

…rting for StoreAPI + Optimized iter chunk dedup. (#2710)

* Deduplicate chunk dups on proxy StoreAPI level. Recommend chunk sorting for StoreAPI.

Also: Merge same series together on proxy level instead select. This allows better dedup efficiency.

Partially fixes: #2303

Cases like overlapped data from store and sidecar and 1:1 duplicates are optimized as soon as it's possible.
This case was highly visible on GitLab repro data and exists in most of Thanos setup.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Optimized algorithm to combine series only on start.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Optimized chunk comparision for overlaps.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Optimized deduplication for deduplicated chunk on query level as well.

Never use proto .String() in fast path!

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
# Conflicts:
#	CHANGELOG.md
#	pkg/store/storepb/custom.go
#	pkg/store/storepb/custom_test.go
@brancz brancz merged commit bb16dbd into release-0.13 Jun 4, 2020
@brancz brancz deleted the optimize-store-0.13 branch June 4, 2020 08:47
@bwplotka
Copy link
Member Author

bwplotka commented Jun 4, 2020

We seen huge improvement on our setup:
image

@pracucci
Copy link
Contributor

pracucci commented Jun 4, 2020

We seen huge improvement on our setup:

What does up_custom_query_last_duration tracks?

@brancz
Copy link
Member

brancz commented Jun 4, 2020

Basically it runs heavy analytical queries against any Prometheus API endpoint.

paulfantom added a commit to paulfantom/thanos that referenced this pull request Jul 8, 2020
openshift/master

* upstream/release-0.13:
  Cut release v0.13.0
  shipper: Be strict about upload order unless it's specified so & cut v0.13.0-rc.2 (thanos-io#2765)
  Cut 0.13.0 release. (thanos-io#2762)
  Cut release 0.13.0-rc.1 (thanos-io#2720)
  Store: `irate` and `resets` use now counter downsampling aggregations. (thanos-io#2719)
  deps: Updated minio-go dependency to v6.0.56 to add two region endpoints (thanos-io#2705) (thanos-io#2718)
  store/proxy: Deduplicate chunks on StoreAPI level. Recommend chunk sorting for StoreAPI + Optimized iter chunk dedup. (thanos-io#2710) (thanos-io#2711)
  Allow using multiple memcached clients at the same time. (thanos-io#2648) (thanos-io#2698)
  Updated Prometheus as little as possible to include Isolation fix. (thanos-io#2697)
  Release fix attempt2.
  Fixed test job. (thanos-io#2650)
  Fixed promu build to build in compatible directory that crossbuild understands.
  Cut v0.13.0-rc.0 (thanos-io#2628)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants