Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Print memory statistics during task ending when leak is found #6959

Merged
merged 1 commit into from
Aug 21, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions gluten-data/src/main/scala/org/apache/gluten/runtime/Runtime.scala
Original file line number Diff line number Diff line change
Expand Up @@ -97,16 +97,17 @@ object Runtime {
throw new GlutenException(
s"Runtime instance already released: $handle, ${resourceName()}, ${priority()}")
}

def dump(): KnownNameAndStats = {
new KnownNameAndStats() {
override def name: String = resourceName()
override def stats: MemoryUsageStats = collectMemoryUsage()
}
}

if (LOGGER.isDebugEnabled) {
LOGGER.debug(
SparkMemoryUtil.prettyPrintStats(
"About to release memory manager, usage dump:",
new KnownNameAndStats() {
override def name: String = resourceName()

override def stats: MemoryUsageStats = collectMemoryUsage()
}
))
SparkMemoryUtil.prettyPrintStats("About to release memory manager, usage dump:", dump()))
}

RuntimeJniWrapper.releaseRuntime(handle)
Expand All @@ -115,10 +116,11 @@ object Runtime {
LOGGER.warn(
String.format(
"%s Reservation listener %s still reserved non-zero bytes, which may cause memory" +
" leak, size: %s. ",
" leak, size: %s, dump: %s ",
name,
rl.toString,
SparkMemoryUtil.bytesToString(rl.getUsedBytes)
SparkMemoryUtil.bytesToString(rl.getUsedBytes),
dump()
))
}
}
Expand Down
Loading