Add memory profiling / logging #1701
Description
What is the problem the feature request solves?
I would like to add a config that enables memory profiling so that we can monitor JVM and native memory usage throughout the lifetime of a Spark session or job. This data should be written out in a structured file format from which we can generate charts.
In JVM side, we can use:
val memoryMXBean = ManagementFactory.getMemoryMXBean
val heap = memoryMXBean.getHeapMemoryUsage
val nonHeap = memoryMXBean.getNonHeapMemoryUsageIn native side, we can use the procfs crate:
let pid = std::process::id();
let process = Process::new(pid as i32).unwrap();
let statm = process.statm().unwrap();By logging JVM usage and overall process memory information, we can infer how much native memory is used. We can also log how much memory is reserved in the native memory pools and start to see how that aligns with actual usage.
Describe the potential solution
No response
Additional context
No response