query: Deduplication latency

[yineima]

thanos version
v0.9.0

prometheus version
version 2.14.0

Object Storage Provider:
no use of object store，use local disk

What happened:
After useing query.replica-label for HA, lately I found queries consume much more time than before
deduplication makes query slow
same query，not query range
deduplication cost 23s, however no deduplication only cost 293ms.
this is a huge cost for HA

here is my Screenshot

the query is sum(rate(appnews_importable_bf_request_num{job="import_interface",product="Inews"} [1m] offset 24h))

appnews_importable_bf_request_num{job="import_interface",product="Inews"} offset 24h has about 1200 samples

I wonder Does it make sense？
or Is there any wrong with my usage？

What you expected to happen:
not too much worse proformace than before

hope you could reply this issue ASAP
this issus makes me crazy.
Thank you.

[bwplotka]

Thanks for this report.

So normally the overhead is unnoticeable, so there might be some bug there for your case here. It would be nice to dive deeper. What I find really suspicious is that appnews_importable_bf_request_num{job="import_interface",product="Inews"} offset 24h takes 322ms WITH Deduplication, but sum(rate(appnews_importable_bf_request_num{job="import_interface",product="Inews"} [1m] offset 24h) is suddenly 24s? That's kind of unlikely.

Let's dive deeper:

1. You say 1200 samples. But this is for instant query without range selector [1m]. Can we do instant query on console for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h? If you have 1200 series I expect it to be something like 4x1200 samples.
2. Can you check latency for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h with and without deduplictation?

[bwplotka]

Also, Try without offset, but I think the issue that we are selecting all 24h+1m was fixed in v0.9.0 already.

[yineima]

Thanks for this report.

So normally the overhead is unnoticeable, so there might be some bug there for your case here. It would be nice to dive deeper. What I find really suspicious is that appnews_importable_bf_request_num{job="import_interface",product="Inews"} offset 24h takes 322ms WITH Deduplication, but sum(rate(appnews_importable_bf_request_num{job="import_interface",product="Inews"} [1m] offset 24h) is suddenly 24s? That's kind of unlikely.

Let's dive deeper:

1. You say 1200 samples. But this is for instant query without range selector [1m]. Can we do instant query on console for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h? If you have 1200 series I expect it to be something like 4x1200 samples.
2. Can you check latency for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h with and without deduplictation?

Thank you for your quick reply.

1. with range selector [1m]. one serie could have 4 or 5 samples.
   So when I do instant query on console for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h with deduplication, I got this
   

2.this is the latency comparsion for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m] offset 24h

3.this is the latency comparsion for appnews_importable_bf_request_num{job="import_interface",product="Inews"}[1m]

Thank you!

[yineima]

Does it may have some connection with query traffic?
I have set --query.timeout 20m --query.max-concurrent 200 for query
but I've not found any config of concurrent for sidecar

[bwplotka]

I mean try to not use the sidecar for anything else for period of tests (:

[daixiang0]

Do we have a tool or something to do pre-deduplication? If not, would be implemented it in compactor?

[bwplotka]

Yes. Please see: #1014 (:

…

On Fri, 20 Mar 2020 at 02:47, Xiang Dai ***@***.***> wrote: Do we have a tool or something to do pre-deduplication? If not, would be implemented it in compactor? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O4TVOJ3OXZCDZD5N6LRILKLHANCNFSM4LC7SF5A> .

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[PowerSurj]

This seems still an issue. Deduplication adds significant latencies, particularly visible when queries have big number of series.