Add support for different kinds of memory (HBM, FAM, and others)

[naveen-rn]

Goal

Allow explicit allocation of symmetric heap on different kinds of memory available in the system architecture.

Problem

Emerging systems support multiple kinds of memory with different performance and capacity characteristics. At present, the symmetric heap is created on a memory location determined by the implementation and the users don't have options to select the kind of memory for allocating the symmetric heap.

Proposed Solution - 1 - Environment Variable based Allocation

We propose few runtime changes to allow the users to determine the memory location at which the symmetric heaps can be created and call these user-determined memory locations as Memory Partitions. Each Memory Partitions feature a list of traits to define their characteristics and kind of memory being one of those featured traits.

More details on the proposed changes, HW/SW Impact analysis are available in Redmine Ticket and in this OpenSHMEM-17 workshop paper.

Example

# Environmental variable
SHMEM_SYMMETRIC_PARTITION<ID>=size=<size>[:pgsize=<page_size>][:kind=<mem_kind>][:policy=<mem_policy>]

# C/C++ routines
void *shmem_partition_malloc(size_t size, int partition_id)
void *shmem_partition_align(size_t alignment, size_t size, int partition_id)

Proposed Solution - 2 - OpenSHMEM Teams based memory allocation

This solution is still under discussion. Will be updated once the OpenSHMEM Teams proposal is complete and ready for adding into the specification.

Notes

Symmetric Memory Partition is a feature for allocating SHEAP on different memory kinds explicitly by the users. There are multiple ways to achieve it. Initially it started as env-variable based approach and later evolved into team-based approach. So, the feature itself isn't fixed on any one particular means to achieve the goal. Even both the solution can be adopted.

Blocked by Teams proposal for further discussioin

[naveen-rn]

Initial draft specification based on OpenSHMEM-1.3 specification. Needs update.

mem_partition_initial_draft.pdf

[naveen-rn]

As the teams working group is working on a proposal to introduce PE subsets in OpenSHMEM. The plan is to move towards the team-based symmetric heap. This issue should be an initial starting point for our discussions.

[naveen-rn]

Linking related documents: https://gist.github.com/nspark/b36fd446252ed789005223612808f45e

[naveen-rn]

Proposed Solution - 2 - OpenSHMEM Teams based memory allocation

Requirement

Similar to Proposed Solution - 1, we need to support different kinds of memory including HBM, fabric attached memory, low latency memory and other possible types. While the proposed solution - 1 is simple, it has limited usage model. This solution provides extensive and flexible support for utilizing different types of memory by relating teams, team-based contexts, and team-based symmetric heaps.

Background:

1. Contexts are essentially a team-object with the new OpenSHMEM Teams proposal.
2. There are two types of contexts:

• Contexts created from the global team
• Contexts created from the user-created teams

Proposal:

1. Symmetric memory created with different traits like pagesize, memory type, size, and other properties using a new collective API shmem_create_spaces() are called Memory Spaces.
2. Spaces are created on a specific team-object.

• Spaces have many-to-one mapping to teams, at most one team per space.
• Multiple spaces can be created on a single team.

3. Contexts are expected to be mapped to a space through the teams.

• Contexts have many-to-one mapping to spaces, at most one space is mapped to a context.
• Multiple contexts sharing the same team objects can be mapped to different spaces, but should not break the previous requirement
• Multiple contexts sharing the same team object can be mapped to the same space.

API:

shmem_create_spaces(IN shmem_team_t team, IN shmem_config_space_t traits, OUT shmem_space_t *space);
shmem_ctx_set_space(IN shmem_space_t space, IN shmem_ctx_t ctx);

basic traits available for creating spaces are type of memory (HBM, LOW_LATENCY, DDR), pagesize, size in bytes, memory policy.

Compatibility:

No backward compatibility issues with the current specification.

Under Consideration:

Do we need new explicit context option to decide on the memory type to be used, say SHMEM_CTX_FAM for specifying whether the context created would be then mapped to memory space created on an external memory/FAM? This can also be done internally during the context mapping operation with the spaces.