-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Negative memory causes _cat APIs to fail #4474
Comments
The root cause is the calculation here: OpenSearch/server/src/main/java/org/opensearch/indices/IndicesQueryCache.java Lines 117 to 139 in 10bff0c
In the edge case where Then later in calculating total memory, the max long value (line 509) is added to other fields, resulting in overflow. OpenSearch/server/src/main/java/org/opensearch/action/admin/indices/stats/CommonStats.java Lines 503 to 516 in ad1c803
Based on the comment about distributing shared ram usage, I suspect that the value of |
Good debugging @dbwiddis :) let’s see a fix? |
Alas, while I can backtrace call hierarchies like a pro, I'm not at all familiar with this section of the code. Happy to submit a PR if someone can tell me the best resolution. Always 0? Max out the |
@dbwiddis I think your last note about the deeper bug elsewhere is likely a cause. From what I can tell, in this particular flow there are shards present (hence the call to |
While fixing this I believe I isolated the root cause: OpenSearch/server/src/main/java/org/opensearch/indices/IndicesQueryCache.java Lines 198 to 204 in 10bff0c
We are in a situation where all documents have been removed from the shard but the cache still retains cached filters from previous queries. This comment (or a variant) appears in |
A reproducing test case already existed in |
Just encountered this on a cluster running 3 opensearch version 1.3.11 nodes and one 1.3.8 node ( in middle of failover to the newer version) ; clearing caches fixed the error. |
Describe the bug
_cat/indices API is failing with below error:
Found similar issue in Elasticsearch repo elastic/elasticsearch#55434
To Reproduce
Don't have a clean way to reproduce this.
Expected behavior
_cat APIs should fail with the error described.
The text was updated successfully, but these errors were encountered: