-
Notifications
You must be signed in to change notification settings - Fork 318
Added script that will collect thread and heap dumps in case test timeout on CI. #9414
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
77 commits
Select commit
Hold shift + click to select a range
db67d8e
Debug freezing test on CI.
AlexeyKuznetsov-DD b4d7fd6
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 025e232
Trying to reproduce.
AlexeyKuznetsov-DD b62fc03
Trying to reproduce.
AlexeyKuznetsov-DD d6d4454
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 835f313
Trying to reproduce.
AlexeyKuznetsov-DD 5892ac5
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 7cafe75
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 01b34c3
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD a1fae6a
Updated flaky tests.
AlexeyKuznetsov-DD 375a9ff
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 60deecd
Trying to reproduce.
AlexeyKuznetsov-DD 0a415ca
Trying to reproduce.
AlexeyKuznetsov-DD 4cfbfd7
Merge branch 'alexeyk/apache-http-5-flaky-tests' into alexeyk/debug-c…
AlexeyKuznetsov-DD 7a263b9
Trying to reproduce.
AlexeyKuznetsov-DD fac9597
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 26d7a12
Trying to reproduce.
AlexeyKuznetsov-DD 9179563
Trying to reproduce.
AlexeyKuznetsov-DD 16cf188
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 435d111
Trying to reproduce.
AlexeyKuznetsov-DD fdb23a4
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 9232258
Trying to fix with `useStrictTraceWrites == false`
AlexeyKuznetsov-DD 2c0c4ce
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 118a0ef
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD d6ce19f
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD ba2a706
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD 6f7706b
Trying to reproduce.
AlexeyKuznetsov-DD b8f1dd3
Rolled back `useStrictTraceWrites == false`
AlexeyKuznetsov-DD 4429e5d
Refactored dump logic: start thread in advance and sleep.
AlexeyKuznetsov-DD ce809d7
Trying to reproduce
AlexeyKuznetsov-DD 79204d9
Refactored thread dump logic
AlexeyKuznetsov-DD d777c3c
Refactored thread dump logic
AlexeyKuznetsov-DD dd2ea8c
Trying to reproduce
AlexeyKuznetsov-DD b0de3f5
Merge branch 'master' into alexeyk/debug-ci-freeze
AlexeyKuznetsov-DD bb06758
Attempt to get dump via gradle
AlexeyKuznetsov-DD 556e780
Fixed codenarc.
AlexeyKuznetsov-DD f55ccb5
Trying to reproduce.
AlexeyKuznetsov-DD 2d4aaa6
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD f2ec317
Trying to reproduce.
AlexeyKuznetsov-DD 161991a
Trying to reproduce with heap dump.
AlexeyKuznetsov-DD cd1c49c
Skip non-gradle heap dumps
AlexeyKuznetsov-DD 331e308
Trying to reproduce.
AlexeyKuznetsov-DD a627099
Trying to reproduce.
AlexeyKuznetsov-DD 4e2c0eb
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD dbe7361
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 255e66c
Trying to reproduce.
AlexeyKuznetsov-DD 5ebc09c
Limit Heap dumps only for Gradle Test Executors.
AlexeyKuznetsov-DD e26f582
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 19e5450
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 714fdb9
Minor refactoring.
AlexeyKuznetsov-DD f382692
Minor refactoring.
AlexeyKuznetsov-DD be9dae2
Minor refactoring.
AlexeyKuznetsov-DD d920e92
Fixed typo
AlexeyKuznetsov-DD a927c0b
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 70f5de0
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD fdfbadd
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD 563ffce
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 27d30a6
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 7d7f3c1
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD 068d9b4
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD 82797f5
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD 71ebf30
Trying to reproduce freeze on CI.
AlexeyKuznetsov-DD 90bff07
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 62fe342
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD d69ee1c
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD a51977e
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 4f026d1
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 65dbc52
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 05bebd1
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD a426b8e
Moved dump logic to separate script. Refactored to be ready to merge …
AlexeyKuznetsov-DD 3b64b91
Revert changes that should be applied as separate PR.
AlexeyKuznetsov-DD 01960d3
Revert changes that should be applied as separate PR.
AlexeyKuznetsov-DD 518d93f
Minor cleanup.
AlexeyKuznetsov-DD 32a2665
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 27a9f4b
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD 9fb4159
Merge branch 'master' into alexeyk/debug-ci-freeze-gradle
AlexeyKuznetsov-DD f61c7bc
Fixed review comments.
AlexeyKuznetsov-DD File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| import java.util.concurrent.Executors | ||
| import java.util.concurrent.TimeUnit | ||
|
|
||
| // Schedule thread and heap dumps collection near test timeout. | ||
| tasks.withType(Test).configureEach { testTask -> | ||
| doFirst { | ||
| def scheduler = Executors.newSingleThreadScheduledExecutor({ r -> | ||
| Thread t = new Thread(r, 'dump-scheduler') | ||
| t.daemon = true | ||
| t | ||
| }) | ||
|
|
||
| // Calculate delay for taking dumps as test timeout minus 2 minutes, but no less than 1 minute. | ||
| def delayMinutes = Math.max(1L, timeout.get().minusMinutes(2).toMinutes()) | ||
|
|
||
| def future = scheduler.schedule({ | ||
| try { | ||
| // Use Gradle's build dir and adjust for CI artifacts collection if needed. | ||
| def dumpDir = layout.buildDirectory.dir('dumps').map { | ||
| if (providers.environmentVariable("CI").isPresent()) { | ||
| // Move reports into the folder collected by the collect_reports.sh script. | ||
| new File(it.getAsFile().absolutePath.replace('dd-trace-java/dd-java-agent', 'dd-trace-java/workspace/dd-java-agent')) | ||
| } else { | ||
| it.asFile | ||
| } | ||
| }.get() | ||
|
|
||
| dumpDir.mkdirs() | ||
|
|
||
| // Collect PIDs of all Java processes. | ||
| def jvmProcesses = 'jcmd -l'.execute().text.readLines() | ||
|
|
||
| // Collect pids for 'Gradle test executors'. | ||
| def pids = jvmProcesses | ||
| .findAll({ it.contains('Gradle Test Executor') }) | ||
| .collect({ it.substring(0, it.indexOf(' ')) }) | ||
|
|
||
| pids.each { pid -> | ||
| logger.warn("Taking dumps for: ${testTask.getPath()}") | ||
|
|
||
| // Collect thread dump. | ||
| def threadDumpFile = new File(dumpDir, "${pid}-thread-dump-${System.currentTimeMillis()}.log") | ||
AlexeyKuznetsov-DD marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| new ProcessBuilder('jcmd', pid, 'Thread.print', '-l') | ||
| .redirectErrorStream(true) | ||
| .redirectOutput(threadDumpFile) | ||
| .start() | ||
| .waitFor() | ||
|
|
||
| // Collect heap dump. | ||
| def heapDumpFile = new File(dumpDir, "${pid}-heap-dump-${System.currentTimeMillis()}.hprof").absolutePath | ||
| def cmd = "jcmd ${pid} GC.heap_dump ${heapDumpFile}" | ||
| cmd.execute().waitFor() | ||
| } | ||
| } catch (Throwable e) { | ||
| logger.warn("Dumping failed: ${e.message}") | ||
| } | ||
| finally { | ||
| scheduler.shutdown() | ||
| } | ||
AlexeyKuznetsov-DD marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }, delayMinutes, TimeUnit.MINUTES) | ||
|
|
||
| // Store handles for cancellation in doLast. | ||
| ext.dumpFuture = future | ||
| ext.dumpScheduler = scheduler | ||
| } | ||
|
|
||
| doLast { | ||
| // Cancel if the task finished before the scheduled dump. | ||
| try { | ||
| ext.dumpFuture?.cancel(false) | ||
| } finally { | ||
| ext.dumpScheduler?.shutdownNow() | ||
| } | ||
| } | ||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.