Adding crash tracking to integration tests #7855
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dougqh
added
run-tests: ibm8
dougqh
changed the title
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 7 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.089 s) : 0, 1088681
Total [baseline] (10.48 s) : 0, 10480299
Agent [candidate] (1.093 s) : 0, 1092546
Total [candidate] (10.423 s) : 0, 10423428
section appsec
Agent [baseline] (1.224 s) : 0, 1224115
Total [baseline] (10.854 s) : 0, 10853981
Agent [candidate] (1.224 s) : 0, 1224256
Total [candidate] (10.728 s) : 0, 10728475
section iast
Agent [baseline] (1.226 s) : 0, 1225895
Total [baseline] (10.997 s) : 0, 10997149
Agent [candidate] (1.218 s) : 0, 1218072
Total [candidate] (10.936 s) : 0, 10935622
section profiling
Agent [baseline] (1.315 s) : 0, 1315161
Total [baseline] (10.839 s) : 0, 10838864
Agent [candidate] (1.328 s) : 0, 1328200
Total [candidate] (10.873 s) : 0, 10872549
gantt
title petclinic - break down per module: candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.202 ms) : 0, 692202
BytebuddyAgent [candidate] (693.346 ms) : 0, 693346
GlobalTracer [baseline] (316.875 ms) : 0, 316875
GlobalTracer [candidate] (317.272 ms) : 0, 317272
AppSec [baseline] (54.645 ms) : 0, 54645
AppSec [candidate] (54.767 ms) : 0, 54767
Remote Config [baseline] (675.434 µs) : 0, 675
Remote Config [candidate] (686.166 µs) : 0, 686
Telemetry [baseline] (10.632 ms) : 0, 10632
Telemetry [candidate] (12.734 ms) : 0, 12734
section appsec
BytebuddyAgent [baseline] (710.781 ms) : 0, 710781
BytebuddyAgent [candidate] (710.767 ms) : 0, 710767
GlobalTracer [baseline] (313.714 ms) : 0, 313714
GlobalTracer [candidate] (314.134 ms) : 0, 314134
AppSec [baseline] (166.83 ms) : 0, 166830
AppSec [candidate] (167.057 ms) : 0, 167057
IAST [baseline] (19.87 ms) : 0, 19870
IAST [candidate] (19.001 ms) : 0, 19001
Remote Config [baseline] (653.694 µs) : 0, 654
Remote Config [candidate] (637.726 µs) : 0, 638
Telemetry [baseline] (8.554 ms) : 0, 8554
Telemetry [candidate] (8.462 ms) : 0, 8462
section iast
BytebuddyAgent [baseline] (816.923 ms) : 0, 816923
BytebuddyAgent [candidate] (812.043 ms) : 0, 812043
GlobalTracer [baseline] (307.67 ms) : 0, 307670
GlobalTracer [candidate] (305.452 ms) : 0, 305452
AppSec [baseline] (58.402 ms) : 0, 58402
AppSec [candidate] (57.687 ms) : 0, 57687
IAST [baseline] (21.016 ms) : 0, 21016
IAST [candidate] (20.975 ms) : 0, 20975
Remote Config [baseline] (637.365 µs) : 0, 637
Remote Config [candidate] (637.627 µs) : 0, 638
Telemetry [baseline] (7.543 ms) : 0, 7543
Telemetry [candidate] (7.547 ms) : 0, 7547
section profiling
BytebuddyAgent [baseline] (688.075 ms) : 0, 688075
BytebuddyAgent [candidate] (697.479 ms) : 0, 697479
GlobalTracer [baseline] (432.967 ms) : 0, 432967
GlobalTracer [candidate] (435.311 ms) : 0, 435311
AppSec [baseline] (53.537 ms) : 0, 53537
AppSec [candidate] (53.99 ms) : 0, 53990
Remote Config [baseline] (662.238 µs) : 0, 662
Remote Config [candidate] (662.795 µs) : 0, 663
Telemetry [baseline] (7.657 ms) : 0, 7657
Telemetry [candidate] (7.737 ms) : 0, 7737
ProfilingAgent [baseline] (93.263 ms) : 0, 93263
ProfilingAgent [candidate] (93.384 ms) : 0, 93384
Profiling [baseline] (93.286 ms) : 0, 93286
Profiling [candidate] (93.407 ms) : 0, 93407
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.094 s) : 0, 1094296
Total [baseline] (8.673 s) : 0, 8673358
Agent [candidate] (1.091 s) : 0, 1091121
Total [candidate] (8.655 s) : 0, 8654575
section iast
Agent [baseline] (1.216 s) : 0, 1215964
Total [baseline] (9.201 s) : 0, 9200833
Agent [candidate] (1.235 s) : 0, 1235477
Total [candidate] (9.238 s) : 0, 9238204
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.216 s) : 0, 1215932
Total [baseline] (9.151 s) : 0, 9151082
Agent [candidate] (1.216 s) : 0, 1215989
Total [candidate] (9.169 s) : 0, 9169400
section iast_TELEMETRY_OFF
Agent [baseline] (1.215 s) : 0, 1214557
Total [baseline] (9.169 s) : 0, 9168946
Agent [candidate] (1.215 s) : 0, 1214633
Total [candidate] (9.175 s) : 0, 9174797
gantt
title insecure-bank - break down per module: candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (696.202 ms) : 0, 696202
BytebuddyAgent [candidate] (694.615 ms) : 0, 694615
GlobalTracer [baseline] (317.42 ms) : 0, 317420
GlobalTracer [candidate] (318.121 ms) : 0, 318121
AppSec [baseline] (54.861 ms) : 0, 54861
AppSec [candidate] (54.858 ms) : 0, 54858
Remote Config [baseline] (684.781 µs) : 0, 685
Remote Config [candidate] (688.247 µs) : 0, 688
Telemetry [baseline] (11.404 ms) : 0, 11404
Telemetry [candidate] (9.116 ms) : 0, 9116
section iast
BytebuddyAgent [baseline] (810.48 ms) : 0, 810480
BytebuddyAgent [candidate] (824.462 ms) : 0, 824462
GlobalTracer [baseline] (304.65 ms) : 0, 304650
GlobalTracer [candidate] (309.801 ms) : 0, 309801
AppSec [baseline] (57.312 ms) : 0, 57312
AppSec [candidate] (58.202 ms) : 0, 58202
IAST [baseline] (21.64 ms) : 0, 21640
IAST [candidate] (20.789 ms) : 0, 20789
Remote Config [baseline] (645.984 µs) : 0, 646
Remote Config [candidate] (629.282 µs) : 0, 629
Telemetry [baseline] (7.518 ms) : 0, 7518
Telemetry [candidate] (7.543 ms) : 0, 7543
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (809.965 ms) : 0, 809965
BytebuddyAgent [candidate] (809.809 ms) : 0, 809809
GlobalTracer [baseline] (305.496 ms) : 0, 305496
GlobalTracer [candidate] (305.361 ms) : 0, 305361
AppSec [baseline] (57.693 ms) : 0, 57693
AppSec [candidate] (57.913 ms) : 0, 57913
IAST [baseline] (20.921 ms) : 0, 20921
IAST [candidate] (21.04 ms) : 0, 21040
Remote Config [baseline] (645.74 µs) : 0, 646
Remote Config [candidate] (637.672 µs) : 0, 638
Telemetry [baseline] (7.517 ms) : 0, 7517
Telemetry [candidate] (7.538 ms) : 0, 7538
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (808.816 ms) : 0, 808816
BytebuddyAgent [candidate] (808.162 ms) : 0, 808162
GlobalTracer [baseline] (305.455 ms) : 0, 305455
GlobalTracer [candidate] (306.957 ms) : 0, 306957
AppSec [baseline] (57.238 ms) : 0, 57238
AppSec [candidate] (57.62 ms) : 0, 57620
IAST [baseline] (21.336 ms) : 0, 21336
IAST [candidate] (20.202 ms) : 0, 20202
Remote Config [baseline] (620.345 µs) : 0, 620
Remote Config [candidate] (601.522 µs) : 0, 602
Telemetry [baseline] (7.428 ms) : 0, 7428
Telemetry [candidate] (7.364 ms) : 0, 7364
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section baseline
no_agent (1.36 ms) : 1340, 1379
. : milestone, 1360,
appsec (1.756 ms) : 1731, 1782
. : milestone, 1756,
appsec_no_iast (1.764 ms) : 1739, 1788
. : milestone, 1764,
iast (1.505 ms) : 1483, 1527
. : milestone, 1505,
profiling (1.578 ms) : 1553, 1602
. : milestone, 1578,
tracing (1.509 ms) : 1482, 1537
. : milestone, 1509,
section candidate
no_agent (1.363 ms) : 1343, 1382
. : milestone, 1363,
appsec (1.76 ms) : 1735, 1785
. : milestone, 1760,
appsec_no_iast (1.747 ms) : 1722, 1772
. : milestone, 1747,
iast (1.501 ms) : 1478, 1524
. : milestone, 1501,
profiling (1.585 ms) : 1560, 1609
. : milestone, 1585,
tracing (1.498 ms) : 1474, 1523
. : milestone, 1498,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section baseline
no_agent (376.943 µs) : 357, 397
. : milestone, 377,
iast (505.636 µs) : 483, 528
. : milestone, 506,
iast_FULL (659.517 µs) : 638, 681
. : milestone, 660,
iast_GLOBAL (521.62 µs) : 500, 543
. : milestone, 522,
iast_HARDCODED_SECRET_DISABLED (495.82 µs) : 475, 517
. : milestone, 496,
iast_INACTIVE (457.859 µs) : 437, 479
. : milestone, 458,
iast_TELEMETRY_OFF (485.01 µs) : 463, 507
. : milestone, 485,
tracing (451.093 µs) : 431, 472
. : milestone, 451,
section candidate
no_agent (381.114 µs) : 361, 401
. : milestone, 381,
iast (507.218 µs) : 484, 530
. : milestone, 507,
iast_FULL (658.829 µs) : 637, 680
. : milestone, 659,
iast_GLOBAL (521.404 µs) : 500, 543
. : milestone, 521,
iast_HARDCODED_SECRET_DISABLED (494.051 µs) : 473, 515
. : milestone, 494,
iast_INACTIVE (453.189 µs) : 432, 474
. : milestone, 453,
iast_TELEMETRY_OFF (481.536 µs) : 460, 503
. : milestone, 482,
tracing (451.67 µs) : 431, 473
. : milestone, 452,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section baseline
no_agent (1.467 ms) : 1455, 1478
. : milestone, 1467,
appsec (2.335 ms) : 2293, 2376
. : milestone, 2335,
iast (2.091 ms) : 2038, 2144
. : milestone, 2091,
iast_GLOBAL (2.131 ms) : 2078, 2185
. : milestone, 2131,
profiling (1.952 ms) : 1910, 1994
. : milestone, 1952,
tracing (1.929 ms) : 1889, 1970
. : milestone, 1929,
section candidate
no_agent (1.467 ms) : 1456, 1479
. : milestone, 1467,
appsec (2.346 ms) : 2304, 2387
. : milestone, 2346,
iast (2.08 ms) : 2027, 2132
. : milestone, 2080,
iast_GLOBAL (2.12 ms) : 2068, 2173
. : milestone, 2120,
profiling (1.965 ms) : 1922, 2008
. : milestone, 1965,
tracing (1.92 ms) : 1880, 1960
. : milestone, 1920,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.44.0-SNAPSHOT~e97ef02b4c, baseline=1.44.0-SNAPSHOT~896dd6b380
dateFormat X
axisFormat %s
section baseline
no_agent (14.954 s) : 14954000, 14954000
. : milestone, 14954000,
appsec (14.852 s) : 14852000, 14852000
. : milestone, 14852000,
iast (18.79 s) : 18790000, 18790000
. : milestone, 18790000,
iast_GLOBAL (17.778 s) : 17778000, 17778000
. : milestone, 17778000,
profiling (15.013 s) : 15013000, 15013000
. : milestone, 15013000,
tracing (15.021 s) : 15021000, 15021000
. : milestone, 15021000,
section candidate
no_agent (15.242 s) : 15242000, 15242000
. : milestone, 15242000,
appsec (15.041 s) : 15041000, 15041000
. : milestone, 15041000,
iast (18.906 s) : 18906000, 18906000
. : milestone, 18906000,
iast_GLOBAL (17.702 s) : 17702000, 17702000
. : milestone, 17702000,
profiling (15.509 s) : 15509000, 15509000
. : milestone, 15509000,
tracing (14.953 s) : 14953000, 14953000
. : milestone, 14953000,
|
dougqh
changed the title
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
dougqh
commented
Nov 19, 2024
| @@ -115,7 +115,17 @@ public static void waitForPortToOpen( | |||
| } | |||
|
|
|||
| if (!process.isAlive()) { | |||
| throw new RuntimeException("Process died before port " + port + " was opened"); | |||
There was a problem hiding this comment.
I made various improvements to the harness itself to make debugging process that exit abnormally easier
dougqh
commented
Nov 19, 2024
| @@ -22,6 +22,16 @@ abstract class AbstractIastServerSmokeTest extends AbstractServerSmokeTest { | |||
| @Shared | |||
| private final JsonSlurper jsonSlurper = new JsonSlurper() | |||
|
|
|||
| @Override | |||
There was a problem hiding this comment.
I have reservations about this.
I haven't yet figured out how to make crash tracking play nice with the tests that build up JAVA_TOOL_OPTIONS. The quoting of the OnError value works as desired when passed as normal argument, but doesn't work when concatenated into a single env var for JAVA_TOOL_OPTIONS.
I'm planning on revisiting this in a subsequent pull request.
dougqh
commented
Nov 19, 2024
|
|
||
| try { | ||
| PortUtils.waitForPortToOpen(port, 240, TimeUnit.SECONDS, process) | ||
| } catch ( Exception e ) { |
There was a problem hiding this comment.
Adding log information to make debug processes failing to start easier for the next person
dougqh
commented
Nov 19, 2024
| try { | ||
| verifyLog(idx, outputFile) | ||
| } catch ( FileNotFoundException e ) { | ||
| if ( !testedProcesses[idx].isAlive() && testedProcesses[idx].exitValue() == 0 ) { |
There was a problem hiding this comment.
As the comment says, suppressing FileNotFoundException from tracer output in the event that the process exited abnormally
It was just making the test output noisy and more confusing
There was a problem hiding this comment.
If the process exited normally but produce no logs, we will no more run isAcceptable to check for the expected output. Isn't an issue?
There was a problem hiding this comment.
Correction; it is doing what I intended. It propagates the exception, but only if the process exited normally.
If the process exited abnormally, then I skip propagate the exception because no output is kind of expected at that point.
I'll this is a questionable change, since Spock does a decent job of the reporting the original issue first. So I'm happy to forego this change if that's what we prefer.
dougqh
commented
Nov 19, 2024
|
|
||
| // DQH - 13 Nov 2024 - Crashtracking bash script doesn't work on OS X, | ||
| // so skipping crash tracking on OS X | ||
| if (addCrashTracking() && !Platform.isJ9() && !Platform.isMac()) { |
There was a problem hiding this comment.
Currently, not adding crash tracking for J9 or Mac.
Once we make the bash script used by crash tracking compatible with Mac's ancient bash, we can revisit this.
There was a problem hiding this comment.
Can you have a look at this PR #7956 to check if it fixes your issue with Mac?
There was a problem hiding this comment.
For J9 we don't have the specific JVM args as well as the J9 crash log parser yet.
There was a problem hiding this comment.
#7956 is merged. Would you like give it a try and check if we can then enable crash tracking on our dev environment too?
There was a problem hiding this comment.
Alright, I'll see if I can enable the mac support now.
5 tasks
|
I cherry-picked #8031 until it gets merged and want to test in CI how it behaves. |
PerfectSlayer
force-pushed
the
dougqh/crash-tracking-in-smoke-tests
branch
from
10f32e7 to
25ae4bb
Compare
PerfectSlayer
requested review from
evanchooly and
juan-fernandez
and removed request for
a team
Debugger benchmarksParameters
See matching parameters
SummaryFound 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.
See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (320.012 µs) : 264, 376
. : milestone, 320,
basic (310.719 µs) : 296, 326
. : milestone, 311,
loop (10.875 ms) : 10827, 10923
. : milestone, 10875,
section candidate
noprobe (304.497 µs) : 276, 333
. : milestone, 304,
basic (309.451 µs) : 299, 320
. : milestone, 309,
loop (10.362 ms) : 10318, 10406
. : milestone, 10362,
|
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
PerfectSlayer
force-pushed
the
dougqh/crash-tracking-in-smoke-tests
branch
from
37ec50d to
2999fe6
Compare
* Excluding J9 * Improving PortUtils to exitCode when process ends abnormally * Improving error reporting when process fails to start Wrapped PortUtils.waitForPortToOpen with exception handling, when exception is raised log file indicating error is added to exception message * Suppressed FileNotException in cleanUpSpec, when process exits abormally since no file is expected and it creates noise * Clarifying exception message * Refining exception suppression, some tests clean-up with test processes still running? * Adding ability to selectively suppress crash tracking * Adding some sanity checks that test processes are up and running as expected * Reworking exception handling logic around verifyLog * Modifying exception handling logic around verifyLog in response to review feedback * When process is alive or exited normally, the exception flows through as is * When process exited abnormally, the exception is wrapped in another exception that indicates the abnormal termination
PerfectSlayer
force-pushed
the
dougqh/crash-tracking-in-smoke-tests
branch
from
2999fe6 to
e97ef02
Compare
|
I fixed any issue with MacOS shell and Other than J9, there is no more exception in place. |
5 tasks
PerfectSlayer
added
the
tag: no release notes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds crash tracking into integration tests
Motivation
Now that single step installer is using crash tracking, want to expand crash tracking coverage in CI
Additional Notes
For now, crash tracking is selectively enabled for a subset of integration tests.
The intention is that future pull requests will broadened the coverage.
Right now, the means of argument construction is incompatible with the tests that pass arguments via JAVA_TOOL_OPTIONS.
IAST tests have also been excluded