More btl uct updates that fix various issues discovered on our hardware #13094
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit fixes a regression I introduced when changing how the endpoint flags are set. There is no guarantee when the remote completion message comes in that the local endpoint has been connected (only the remote endpoint). Instead of always setting the endpoint as ready here the code now only sets that the remote is connected and then sets the endpoint as ready only if both the local and remote endpoints are connected. When the local endpoint is connected it similarly checks that the remote is connected before setting the endpoint as ready. This was verified using RMA-MT with 128 threads and it does fix the race. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
Connection endpoints are created to assist with connection connect-to-endpoint UCT endpoints. In most situations it is created and destroyed for each thread context but it may be used by multiple simultaneous threads. The code retains and releases the endpoint at multiple points where needed. This is overkill. This commit updates the code to retain once per connecting thread (or the original creation) and then releases when the connection is complete. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
devreal
approved these changes
Feb 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes one of the remaining issues I see on our hardware. With these commits it runs cleanly with RMA-MT.
Note the second commit is not really a functional change but a cleanup change.