[FEA] Interface support to reserve an address space without actually allocating it

[teju85]

Describe the solution you'd like
@seunghwak has a very good point here about having a support for reserving memory (aka over-subscription) using CUDA's virtual memory management APIs. I believe this is a good improvement to our existing Allocator, device_buffer and host_buffer interfaces. Thus, filing this issue so that this feature item is not lost.

Additional context
Ref: https://devblogs.nvidia.com/introducing-low-level-gpu-virtual-memory-management/

[github-actions]

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[divyegala]

@teju85 @seunghwak is this something that will be solved by using RMM directly?

[teju85]

Yes, I think we should be able to use RMM's managed_memory_resource for this purpose. But I'd like to hear from @seunghwak if this was what he had in mind or something else.

[seunghwak]

My understanding of this issue is about reserving an address space (https://developer.nvidia.com/blog/introducing-low-level-gpu-virtual-memory-management/) instead of using managed memory.

This address space reservation feature is mainly to avoid reallocation. If we resize a vector, 1) it first allocates the memory block having the new size, 2) copies the old data to the new block, and 3) deallocates the old block. This address reservation feature allows us to replace 1), 2), 3) by just allocating a (new size - old size) block and mapping this to the reserved address space.

Managed memory is to allow using host memory as a lower level buffer (larger in size but slower in speed) for a device memory.

So, these two are two different things, and AFAIK, there was some brief discussion about supporting the address space reservation feature in RMM, but AFAIK, it has not been implemented yet in RMM.

[teju85]

I now remember this discussion. This needs the support of address-reserve API of cuda, which I don't see in RMM. @harrism and/or @jrhemstad is this being planned for in RMM?

[jrhemstad]

Use of the CUDA VMM APIs would be an implementation detail of a memory resource implementation. There aren't any plans to expose explicit RMM APIs for CUDA VMM.

[teju85]

@jrhemstad if there's a use-case like the one Seunghwa discussed above, is there a chance of getting this feature added onto RMM roadmap?

[harrism]

OK, so the specific use case is for resizing buffers, and the reasoning is to avoid copies.

The trouble with this is that RMM implements the memory_resource interface, which originates from std. There is no reallocate in memory_resource (or in C++, for that matter, only C). So adding a feature like this would have to be downstream of the MR interface, which is not attractive. For this reason it has only been discussed for RMM, no decision has been made whether or not to implement it.

So let me ask: is reallocation definitely a bottleneck?

[teju85]

Fair enough @harrism . So far I haven't seen this being the bottleneck, maybe @seunghwak had a use-case in mind when he suggested this feature?

[seunghwak]

So, the biggest benefit of this is memory footprint. We can clearly live without this, but having this will allow more memory footprint optimization for us (and we can handle bigger graphs within the gpu memory limit).

To explain this in more detail,

without this feature, to resize, we need memory size of "old_size + new_size" while with address space reservation, we need only max(old_size, new_size). Say we're doing filtering of multiple blocks, we do something like.

rmm::device_uvector<int> filtered_elements(0, stream);
size_t num_inserted = 0;
for (size_t i = 0; i < num_blocks; ++i) {
  filtered_elements.resize(num_inserted + block_sizes[i], stream);
  filter(...); // num_inserted gets updated...
}
filtered_elements.resize(num_inserted, stream);

So, with the address reservation, we need to just reserve the address space of sum block_sizes[i] but without address reservation, this code will require actual allocation of sum (i=0 to block_sizes -2) block_sizes[i] + sum block_sizes[i] in the worst case (if 100% of the elements passes filtering).

Avoiding copy is a second benefit (but less important as this is pretty fast).