Open
Description
This is a meta issue to track the state of CUDA managed and async memory support.
- Add
cudaMallocAsync
support: done in UCM/CUDA: Add support for CUDA virtual memory withcudaMallocAsync
openucx/ucx#8623, available starting from UCX 1.14+) - Enable CUDA IPC/IB support via UCX staging buffers (
UCX_RNDV_SCHEME=auto UCX_RNDV_FRAG_MEM_TYPE=cuda
): done in Add RMM managed memory argument to benchmarks #836 - Enable CUDA IPC support in UCX without requiring staging buffers: will require UCX protov2
- Enable IB support in UCX without requiring staging buffers: not supportes by CUDA yet
- Fix crashes with non-IB systems when using
UCX_RNDV_FRAG_MEM_TYPE=cuda
: blocked byUCP_REQUEST_FLAG_RNDV_FRAG
assertion failure with endpoint error handling openucx/ucx#8639, will land with UCX protov2 (default in UCX 1.16) - Fix crash with CUDA managed memory (only seems to occur if memory is evicted): blocked by Assertion `req->send.ep == ucp_ep' failed when endpoint/process terminates openucx/ucx#8669
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
No status