CometShuffleMemoryAllocator should use Spark unified memory manager #1064
Description
What is the problem the feature request solves?
Currently Comet JVM shuffle's CometShuffleMemoryAllocator uses a separate unsafe memory allocator and track off-heap memory allocation internally. The memory size is configured by a Comet config.
A better approach should be to use Spark unified memory manager for CometShuffleMemoryAllocator. So we can unify this off-heap memory with Spark off-heap memory config, i.e., we don't need to separately configure Comet JVM shuffle memory.
Describe the potential solution
No response
Additional context
No response