Support for Heap after GC stats #1265

reta · 2021-09-21T17:59:50Z

Signed-off-by: Andriy Redko andriy.redko@aiven.io

Description

JVM stats published today represents the point in time usage of the heap across pools. While
MemoryPoolMXBean::getCollectionUsage provides the memory usage after the Java virtual machine most recently expended effort in recycling unused objects in this memory pool. Exposing this information could be quite helpful while troubleshooting GC and memory issues.

The jvm section of the node statistics had been extended to include the additional details regarding most recent GC cycle for a particular memory pool, under new last_gc_stats section:

            "old" : {                             
              "used_in_bytes" : 50526632,
              "max_in_bytes" : 357957632,
              "peak_used_in_bytes" : 50526632,
              "peak_max_in_bytes" : 357957632,
              "last_gc_stats" : {
                "used_in_bytes" : 42130040,
                "max_in_bytes" : 357957632,
                "usage_percent" : 11
              }                                   
            }

The sample output:

curl http://localhost:9200/_nodes/stats/jvm?pretty

....
{
        ...
        "pools" : {                                                                                                                                                                                      
            "young" : {                                                                                                                                                                                    
              "used_in_bytes" : 103327560,                                                                                                                                                                 
              "max_in_bytes" : 143130624,                                                                                                                                                                  
              "peak_used_in_bytes" : 143130624,                                                                                                                                                            
              "peak_max_in_bytes" : 143130624,                                                                                                                                                             
              "last_gc_stats" : {                                                                                                                                                                          
                "used_in_bytes" : 73248760,                                                                                                                                                                
                "max_in_bytes" : 143130624,                                                                                                                                                                
                "usage_percent" : 51              
              }                                   
            },                                    
            "survivor" : {                        
              "used_in_bytes" : 17431960,         
              "max_in_bytes" : 17891328,          
              "peak_used_in_bytes" : 17891328,    
              "peak_max_in_bytes" : 17891328,     
              "last_gc_stats" : {          
                "used_in_bytes" : 17431960,       
                "max_in_bytes" : 17891328,
                "usage_percent" : 97              
              }                                   
            },                                    
            "old" : {                             
              "used_in_bytes" : 50526632,
              "max_in_bytes" : 357957632,
              "peak_used_in_bytes" : 50526632,
              "peak_max_in_bytes" : 357957632,
              "last_gc_stats" : {
                "used_in_bytes" : 42130040,
                "max_in_bytes" : 357957632,
                "usage_percent" : 11
              }                                   
            }                                     
          }                                       
        },  
        ...

Issues Resolved

#981

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

reta · 2021-09-21T18:00:36Z

@Bukhtawar may I ask you please to take a look, is it something you had mind or there are missing pieces? thank you!

opensearch-ci-bot · 2021-09-21T18:00:54Z

✅ DCO Check Passed 80afb4240ddc4d7330820b41c576b65cba628a55

opensearch-ci-bot · 2021-09-21T18:02:05Z

✅ Gradle Wrapper Validation success 80afb4240ddc4d7330820b41c576b65cba628a55

opensearch-ci-bot · 2021-09-21T18:06:02Z

✅ Gradle Precommit success 80afb4240ddc4d7330820b41c576b65cba628a55

dblock

LGTM

dblock · 2021-09-21T22:36:01Z

start grade check

server/src/main/java/org/opensearch/monitor/jvm/JvmStats.java

Bukhtawar · 2021-09-22T06:23:00Z

server/src/main/java/org/opensearch/monitor/jvm/JvmStats.java

@@ -299,7 +316,9 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
        static final String PEAK_USED_IN_BYTES = "peak_used_in_bytes";
        static final String PEAK_MAX = "peak_max";
        static final String PEAK_MAX_IN_BYTES = "peak_max_in_bytes";
-
+        static final String USAGE_PERCENT = "usage_percent";


Can this be confused with current usage vs usage after GC?

Good question, I think it should not since the field is under last_gc_stats object, in this context, it should not be confusing. Also, the semantic of this field stays consistent between current usage (if we add it there) and usage after GC. What do you think?

opensearch-ci-bot · 2021-09-22T12:24:44Z

❌ DCO Check Failed fd80400f525743b740268d4a1b1db6f6aa41374c
Run ./dev-tools/signoff-check.sh remotes/origin/main fd80400f525743b740268d4a1b1db6f6aa41374c to check locally
Use git commit with -s to add 'Signed-of-by: {EMAIL}' on impacted commits

opensearch-ci-bot · 2021-09-22T12:24:53Z

✅ DCO Check Passed 2c362e40ab4ed1d3fa3af6b7fd00490fab337c1b

opensearch-ci-bot · 2021-09-22T12:24:53Z

✅ Gradle Wrapper Validation success 2c362e40ab4ed1d3fa3af6b7fd00490fab337c1b

opensearch-ci-bot · 2021-09-22T12:32:15Z

✅ Gradle Precommit success 2c362e40ab4ed1d3fa3af6b7fd00490fab337c1b

reta · 2021-09-24T12:16:34Z

@Bukhtawar anything pending for this change from your side? thank you! :-)

Bukhtawar

LGTM.. Thanks @reta for the changes

reta · 2021-09-24T19:45:00Z

@dblock may I ask you please to schedule Gradle check? Thank you!

dblock · 2021-09-24T19:47:44Z

start gradle check

opensearch-ci-bot · 2021-09-24T20:12:07Z

❌ Gradle Check failure 2c362e40ab4ed1d3fa3af6b7fd00490fab337c1b
Log 569

Reports 569

reta · 2021-09-24T20:17:41Z

Gradle Check failure 2c362e4
Log 569

Reports 569

Hm ... Task :distribution:bwc:minor:buildBwcLinuxTar FAILED

> Task :distribution:bwc:staged:buildBwcLinuxTar
 [1.1.0] > Task :distribution:archives:linux-tar:assemble
 [1.1.0] build complete, generating: /var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/distribution/bwc/staged/build/bwc/checkout-1.1/build/569.tar.bz2

> Task :distribution:bwc:minor:buildBwcLinuxTar FAILED
> Task :modules:repository-url:urlFixture#stop

> Task :distribution:bwc:staged:buildBwcLinuxTar
 [1.1.0] 
 [1.1.0] BUILD SUCCESSFUL in 6m 32s
 [1.1.0] 149 actionable tasks: 149 executed

> Task :distribution:bwc:staged:buildBwcLinuxTar FAILED

Same as #1285 ?

reta · 2021-09-27T19:26:37Z

My apologies, @dblock , seems like #1285 was closed, may I ask you please to reschedule the Gradle check whenever you have time, thank you!

dblock · 2021-09-28T12:50:52Z

start gradle check

dblock · 2021-09-28T12:51:23Z

Thanks for hanging in here! I kicked it, but I think this might need a rebase.

opensearch-ci-bot · 2021-09-28T13:18:43Z

❌ Gradle Check failure 2c362e40ab4ed1d3fa3af6b7fd00490fab337c1b
Log 581

Reports 581

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta · 2021-09-28T13:22:03Z

Rebased, thanks a lot @dblock !

opensearch-ci-bot · 2021-09-28T13:25:12Z

✅ DCO Check Passed e7155ca

opensearch-ci-bot · 2021-09-28T13:25:12Z

✅ Gradle Wrapper Validation success e7155ca

opensearch-ci-bot · 2021-09-28T13:36:39Z

✅ Gradle Precommit success e7155ca

dblock · 2021-09-28T16:25:49Z

start gradle check

opensearch-ci-bot · 2021-09-28T16:59:49Z

❌ Gradle Check failure e7155ca
Log 586

Reports 586

reta · 2021-09-28T17:05:54Z

x Gradle Check failure e7155ca Log 586

Reports 586

On it

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

opensearch-ci-bot · 2021-09-28T17:47:15Z

✅ DCO Check Passed 8dfe9a1

opensearch-ci-bot · 2021-09-28T17:47:54Z

✅ Gradle Wrapper Validation success 8dfe9a1

dblock · 2021-09-28T17:55:04Z

start gradle check

opensearch-ci-bot · 2021-09-28T17:56:05Z

✅ Gradle Precommit success 8dfe9a1

opensearch-ci-bot · 2021-09-28T18:35:44Z

✅ Gradle Check success 8dfe9a1
Log 588

Reports 588

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

dblock approved these changes Sep 21, 2021

View reviewed changes

Bukhtawar reviewed Sep 22, 2021

View reviewed changes

server/src/main/java/org/opensearch/monitor/jvm/JvmStats.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Sep 22, 2021

View reviewed changes

server/src/main/java/org/opensearch/monitor/jvm/JvmStats.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Sep 22, 2021

View reviewed changes

reta force-pushed the issues-981 branch from fd80400 to 2c362e4 Compare September 22, 2021 12:20

Bukhtawar approved these changes Sep 24, 2021

View reviewed changes

reta added 2 commits September 28, 2021 09:21

Support for Heap after GC stats

19d145a

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Addressing code review comments

e7155ca

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta force-pushed the issues-981 branch from 2c362e4 to e7155ca Compare September 28, 2021 13:21

Using the right version 2.0.0 (instead of 1.2.0) for the change

8dfe9a1

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

dblock merged commit 80388a8 into opensearch-project:main Sep 28, 2021

reta added a commit to reta/OpenSearch that referenced this pull request Sep 28, 2021

Support for Heap after GC stats (opensearch-project#1265)

fbe0db6

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

reta mentioned this pull request Sep 28, 2021

[Backport] [1.x] Support for Heap after GC stats (opensearch-project#1265) #1309

Merged

5 tasks

tlfeng added enhancement Enhancement or improvement to existing feature or request v1.2.0 Issues related to version 1.2.0 v2.0.0 Version 2.0.0 labels Sep 29, 2021

dblock pushed a commit that referenced this pull request Sep 29, 2021

Support for Heap after GC stats (#1265) (#1309)

37abf3b

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Heap after GC stats #1265

Support for Heap after GC stats #1265

reta commented Sep 21, 2021

reta commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

dblock left a comment

dblock commented Sep 21, 2021

Bukhtawar Sep 22, 2021

reta Sep 22, 2021 •

edited

Loading

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

reta commented Sep 24, 2021

Bukhtawar left a comment

reta commented Sep 24, 2021

dblock commented Sep 24, 2021

opensearch-ci-bot commented Sep 24, 2021

reta commented Sep 24, 2021

reta commented Sep 27, 2021

dblock commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

reta commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

reta commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

Support for Heap after GC stats #1265

Support for Heap after GC stats #1265

Conversation

reta commented Sep 21, 2021

Description

Issues Resolved

Check List

reta commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

opensearch-ci-bot commented Sep 21, 2021

dblock left a comment

Choose a reason for hiding this comment

dblock commented Sep 21, 2021

Bukhtawar Sep 22, 2021

Choose a reason for hiding this comment

reta Sep 22, 2021 • edited Loading

Choose a reason for hiding this comment

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

opensearch-ci-bot commented Sep 22, 2021

reta commented Sep 24, 2021

Bukhtawar left a comment

Choose a reason for hiding this comment

reta commented Sep 24, 2021

dblock commented Sep 24, 2021

opensearch-ci-bot commented Sep 24, 2021

reta commented Sep 24, 2021

reta commented Sep 27, 2021

dblock commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

reta commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

reta commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

dblock commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

opensearch-ci-bot commented Sep 28, 2021

reta Sep 22, 2021 •

edited

Loading