prov/efa: make the peer map a per-endpoint fi_addr-indexed array#12572
Open
charlesstoll wants to merge 1 commit into
Open
prov/efa: make the peer map a per-endpoint fi_addr-indexed array#12572charlesstoll wants to merge 1 commit into
charlesstoll wants to merge 1 commit into
Conversation
The peer map moves from shared efa_conn (a uthash keyed by endpoint) and to efa_rdm_ep as an fi_addr-indexed ofi_dyn_arr of peer pointers. Because the array is per-endpoint and a lookup is a direct index that never mutates shared state, a peer lookup no longer contends with peer creation on another endpoint. This is a structural step toward removing the SRX lock from the fi_send/CQ-read peer lookup; the locking itself is unchanged here. Signed-off-by: Charles Stoll <stollcha@amazon.com>
jiaxiyan
reviewed
Jul 22, 2026
| int32_t efa_rdm_ep_get_peer_ahn(struct efa_rdm_ep *ep, fi_addr_t addr); | ||
| struct efa_rdm_peer *efa_rdm_ep_get_peer_implicit(struct efa_rdm_ep *ep, fi_addr_t addr); | ||
|
|
||
| void efa_rdm_peer_map_init(struct ofi_dyn_arr *arr); |
Contributor
There was a problem hiding this comment.
functions in this file need to start with efa_rdm_ep_*
| { | ||
| struct efa_rdm_peer **slot; | ||
|
|
||
| if (addr > OFI_IDX_MAX_INDEX) |
Contributor
There was a problem hiding this comment.
Do you need this check when ofi_array_at_max have it?
if (index >= max_size || !ofi_array_chunk(arr, index))
return NULL;
Contributor
Author
There was a problem hiding this comment.
This will go away when I rebase on top of the new ofi_dyn_array changes, but one of the apis did not check properly, so I added the bound check there, and it looked weird so I just added the sanity check to all of them for consistency
| "Map entries for fi_addr to peer mapping exhausted.\n"); | ||
| peer = ofi_buf_alloc(ep->efa_rdm_peer_pool); | ||
| if (OFI_UNLIKELY(!peer)) { | ||
| EFA_WARN(FI_LOG_EP_DATA, "Cannot allocate peer\n"); |
| efa_conn_ep_peer_map_insert(conn, map_entry); | ||
|
|
||
| dlist_insert_tail(&map_entry->peer.ep_peer_list_entry, &ep->ep_peer_list); | ||
| if (efa_rdm_peer_map_insert(&ep->fi_addr_to_peer_map, addr, peer)) { |
| @@ -196,13 +201,26 @@ void efa_conn_rdm_deinit(struct efa_av *av, struct efa_conn *conn) | |||
| } | |||
|
|
|||
| assert(ofi_genlock_held(&((struct efa_rdm_domain *) av->domain)->srx_lock)); | |||
| /* since an efa_conn is all connections to a specific remote ep, | ||
| * we must walk all local ep peer maps and remove the connection | ||
| * to the remote ep */ | ||
| dlist_foreach_safe(&av->util_av.ep_list, entry, tmp) { |
Contributor
There was a problem hiding this comment.
ofi_genlock_lock(&av->util_av.ep_list_lock); for ep_list
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The peer map moves from the shared efa_conn (a uthash keyed by endpoint) to efa_rdm_ep as an fi_addr-indexed ofi_dyn_arr of peer pointers. Because the array is per-endpoint and a lookup is a direct index that never mutates shared state, a peer lookup no longer contends with peer creation on another endpoint. This is a structural step toward removing the SRX lock from the fi_send/CQ-read peer lookup; the locking itself is unchanged here.