Querier queries too much data when the head block may have enough data

[dannykopping]

Describe the bug
When executing a long backwards query (> query_ingesters_within), we have the possibility of querying too much data from both the ingesters and the store, particularly if the ingesters' head block contains a large amount of unflushed logs which could satisfy the request.

To Reproduce

• have a simple, non-filtering query like {cluster="us-central1"} | json | line_format "{{.msg}}"
• have a large number of unflushed lines in the head block (~7.5m in our case from all 130+ ingesters)
• execute the query over a 48h time range
• ingesters OOM

Expected behavior
The querier should query the ingesters for their unflushed data, and not need to examine chunks unnecessarily from the store.

Screenshots, Promtail config, or terminal output

Grafana Cloud trace ID: 410e61362f6ac7c2

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[Red-GV]

Hiya @dannykopping,

I've been tinkering with this issue for the past few days and I'm not exactly sure what is the best way for me to fix/test this. I've got two interpretations of "too much data":

1. The data retrieved from the ingester's unflushed data is already beyond the request's limit
2. The data retrieved from the ingester's unflushed data is already beyond the request's time range

and my interpretation of the response to the fix should result in lower requests from Loki to the object storage. Should I open a PR with my current approach to better discuss?

[dannykopping]

I'm not sure this is a problem anymore (this issue is ~a year old) so we should first try to reproduce given the steps above.
The splitting/sharding on the query frontend should be limiting the amount of data that any one ingester is asked to retrieve. In practice we hardly ever see our ingesters OOMing in our cloud.

But to answer your question: #1 is the interpretation that matches what I was thinking when I created this.

[Red-GV]

I've done a bit of testing on my end, and I think I agree that this issue doesn't seem to affect the newer Loki versions. Thanks for your clarification!

[dannykopping]

Ok cool, let's close it until we can reproduce the problem. Thanks @Red-GV!

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[Red-GV]

Hiya @dannykopping,

I've been tinkering with this issue for the past few days and I'm not exactly sure what is the best way for me to fix/test this. I've got two interpretations of "too much data":

1. The data retrieved from the ingester's unflushed data is already beyond the request's limit
2. The data retrieved from the ingester's unflushed data is already beyond the request's time range

and my interpretation of the response to the fix should result in lower requests from Loki to the object storage. Should I open a PR with my current approach to better discuss?

[dannykopping]

I'm not sure this is a problem anymore (this issue is ~a year old) so we should first try to reproduce given the steps above.
The splitting/sharding on the query frontend should be limiting the amount of data that any one ingester is asked to retrieve. In practice we hardly ever see our ingesters OOMing in our cloud.

But to answer your question: #1 is the interpretation that matches what I was thinking when I created this.

[Red-GV]

I've done a bit of testing on my end, and I think I agree that this issue doesn't seem to affect the newer Loki versions. Thanks for your clarification!

[dannykopping]

Ok cool, let's close it until we can reproduce the problem. Thanks @Red-GV!