Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid negative memory result in IndicesQueryCache stats calculation #6917

Merged
merged 3 commits into from
Apr 4, 2023

Conversation

dbwiddis
Copy link
Member

@dbwiddis dbwiddis commented Mar 31, 2023

Description

In the edge case where shardStats is an empty map (caused by removing the last document from an index through a means other than clearing the index), the stats map is also empty. However this case also leads to divide-by-zero calculation which gives a weight of Double.POSITIVE_INFINITY. This rounds to Long.MAX_VALUE for additionalRamBytesUsed for the shard.

Then later in calculating total memory, the max long value is added to other fields, resulting in overflow.

Issues Resolved

Fixes #4474

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dbwiddis dbwiddis added the backport 2.x Backport to 2.x branch label Mar 31, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dbwiddis
Copy link
Member Author

Hmm, seems I made too strong of an assumption on tests:

java.lang.AssertionError: expected:<0> but was:<10480>

Will fix.

@dbwiddis
Copy link
Member Author

Fun thing about that test failure is it let me figure out the root cause of the original bug!

        // This cache stores two things: filters, and doc id sets. At this time
        // we only know that there are no more doc id sets, but we still track
        // recently used queries, which we want to reclaim.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2023

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Daniel Widdis <widdis@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented Apr 1, 2023

Codecov Report

Merging #6917 (a009248) into main (c85d33e) will increase coverage by 0.17%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6917      +/-   ##
============================================
+ Coverage     70.66%   70.83%   +0.17%     
- Complexity    59231    59315      +84     
============================================
  Files          4812     4812              
  Lines        283761   283763       +2     
  Branches      40917    40918       +1     
============================================
+ Hits         200519   201011     +492     
+ Misses        66784    66240     -544     
- Partials      16458    16512      +54     
Impacted Files Coverage Δ
...java/org/opensearch/indices/IndicesQueryCache.java 90.64% <100.00%> (+0.13%) ⬆️

... and 471 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Daniel Widdis <widdis@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2023

Gradle Check (Jenkins) Run Completed with:

@owaiskazi19 owaiskazi19 merged commit c0a8cf8 into opensearch-project:main Apr 4, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1.x 1.x
# Navigate to the new working tree
pushd ../.worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-6917-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c0a8cf8823341cf92b932fe3d12909d427090a2f
# Push it to GitHub
git push --set-upstream origin backport/backport-6917-to-1.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-6917-to-1.x.

@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6917-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c0a8cf8823341cf92b932fe3d12909d427090a2f
# Push it to GitHub
git push --set-upstream origin backport/backport-6917-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6917-to-2.x.

@dbwiddis dbwiddis deleted the shard-stats branch April 4, 2023 00:36
dbwiddis added a commit to dbwiddis/OpenSearch that referenced this pull request Apr 4, 2023
(opensearch-project#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

* Cache stores recent queries so isn't empty even with no doc id sets

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Add test which actually fails without the bug fix

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
@dbwiddis dbwiddis added backport 1.3 Backport to 1.3 branch and removed backport 1.x labels Apr 4, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1.3 1.3
# Navigate to the new working tree
pushd ../.worktrees/backport-1.3
# Create a new branch
git switch --create backport/backport-6917-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c0a8cf8823341cf92b932fe3d12909d427090a2f
# Push it to GitHub
git push --set-upstream origin backport/backport-6917-to-1.3
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-6917-to-1.3.

dbwiddis added a commit to dbwiddis/OpenSearch that referenced this pull request Apr 4, 2023
…pensearch-project#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Cache stores recent queries so isn't empty even with no doc id sets

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Add test which actually fails without the bug fix

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
(cherry picked from commit c0a8cf8)
dbwiddis added a commit to dbwiddis/OpenSearch that referenced this pull request Apr 4, 2023
(opensearch-project#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Cache stores recent queries so isn't empty even with no doc id sets

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Add test which actually fails without the bug fix

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
(cherry picked from commit c0a8cf8)
mitrofmep pushed a commit to mitrofmep/OpenSearch that referenced this pull request Apr 5, 2023
…pensearch-project#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Cache stores recent queries so isn't empty even with no doc id sets

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Add test which actually fails without the bug fix

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Valentin Mitrofanov <mitrofmep@gmail.com>
saratvemulapalli pushed a commit that referenced this pull request Apr 12, 2023
…6966)

(#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Cache stores recent queries so isn't empty even with no doc id sets

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Add test which actually fails without the bug fix

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
(cherry picked from commit c0a8cf8)
owaiskazi19 pushed a commit that referenced this pull request Apr 12, 2023
…6965)

(#6917)

* Avoid negative memory result in IndicesQueryCache stats calculation

* Cache stores recent queries so isn't empty even with no doc id sets



* Add test which actually fails without the bug fix



---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.3 Backport to 1.3 branch backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Negative memory causes _cat APIs to fail
5 participants