CP013: Remove polymorphic allocator

[AerialMantis]

We decided that the polymorphic allocator does fit the requirements we have for affinity-based allocation, so we should drop the pmr_memory_resource_type from the paper and just leave the allocator_type.

[mhoemmen]

@AerialMantis @hcedwar How should we do this in execution_context? Should the memory_resource method still exist, and if so, should it return allocator_type? Or, is the idea that the allocator (instance) behaves as a memory resource? I think you mean the latter, since memory_resource() returns "[a] reference to a polymorphic memory resource of type pmr_memory_resource_type capable of allocating with affinity to the contained resource."

[AerialMantis]

@mhoemmen That's right, originally I was thinking that we remove the pmr_memory_resource_type alias and the memory_resource function and instead just have the allocator_type alias and the allocator function, where the allocator instance would behave as the memory resource. Though we've been discussing this again in light of the decision to separate the execution and memory topologies.

(taken from #52): We decided that for the purposes of affinity-based allocation it didn't make sense to use the pmr memory resource, however after separating the execution and memory topologies, I think the pmr memory resource could be a good choice to represent a node within the memory topology, even if that is then passed to a concrete affinity aware allocator type. Though I think we should (as @hcedwar suggested in #41) make it a type which is derived from the pmr memory resource so that it can be made iterable.

If we were to go down this route perhaps the memory_resource function could return the the highest-level memory resource from the memory topology which the execution resources of the execution context can allocate in. In that case you could construct an instance of allocator_type with any valid memory resource (i.e. any memory resource under that returned from memory_resource) or you could use the allocator instance returned by the allocator function which would be initialised with the memory resource returned from memory_resource.

What do you think? @mhoemmen @hcedwar @Ruyk

[Ruyk]

I think it makes sense. I would like to see something like a memory_context equivalent of the execution_context, but that would not be ready in time for the next meeting.

[mhoemmen]

@AerialMantis Sorry I'm a little late with this -- if we separate execution resource and execution context, then I think it makes sense also to separate memory resource and memory context. This would imply that the default allocator has an implicit memory resource, no?

[AerialMantis]

@mhoemmen @Ruyk sorry I missed this discussion earlier. Yes, I would agree, having a memory context would make sense, particularly in the case where you have discrete memory regions. Perhaps this could also help in supporting agent local storage as it could serve as the context which manages the allocation and lifetime of that memory.

One thought that comes to mind, should a memory context be an allocator type itself such as a pmr allocator, or should it be a separate type which can produce allocators in the same way an execution context can produce executors?

[mhoemmen]

@AerialMantis wrote:

... [S]hould a memory context be an allocator type itself such as a pmr allocator, or should it be a separate type which can produce allocators in the same way an execution context can produce executors?

I think the former. The key distinction for me is between resource and context, not between context and allocator. The point of having a separate "memory resource" comparable to an "execution resource," would be to be able to use a subset of hardware resources for allocations. The "memory resource" would be the subset of hardware, and the "memory (/ allocation) context" would be the memory pool etc.

The pmr allocators conflate context and allocator: one creates the various pool or monotonic buffer resources with an "upstream" memory "resource" (more like a context), but the pool / buffer resources have the same base class as their "upstream" resource.

MPI (the Message Passing Interface for distributed-memory parallel programming) also separates resource and allocator, but conflates context and allocator:

• the memory allocated is an MPI_Win ("shared memory" in a partitioned global address space);
• the "memory resource" is a communicator (MPI_Comm, a subset of parallel processes and their memories); and
• "allocators" consist of functions like MPI_Win_create.

(The "attach a void* allocation to an MPI_Win feature doesn't matter for this discussion.)