Don't use util::generateGid() for MPI messages - #203
Merged
Conversation
csegarragonz
force-pushed
the
no-lock-mpimsgid
branch
from
December 22, 2021 09:05
183eda4 to
867a0da
Compare
Shillaker
requested changes
Dec 22, 2021
Collaborator
There was a problem hiding this comment.
The lock acquired in generateGid should only be acquired once per host (as it's just used to set a single static variable), so I'm very surprised that it would be a source of contention. How do you know it's the lock and not the other stuff that happens in that method? In what sort of workload is that the case?
If there is contention it's most likely that the code is broken somehow. Rather than just dodge the issue for MPI, it would be better to fix the underlying GID bottleneck if there is one. Perhaps we could try switching it to use call_once rather than the mutex-based setup we have at the moment, or use a static initialiser, e.g.
unsigned int generateGid()
{
static size_t gidKeyHash = [](){
std::string gidKey = faabric::util::randomString(GID_LEN);
return std::hash<std::string>{}(gidKey);
} ();
// Existing method
}
Shillaker
approved these changes
Dec 27, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The message ID in the
MPIMessageprotobuf objects is not necessary to guarantee execution integrity. It is just useful for logging and debugging purposes. Further, given that each message has also a sending and receiving rank, a per-rank counter suffices to uniquely identify them. As MPI ranks are single-threaded, we can get away with athread localcounter.Note that, profiling in high contention experiments, showed a non-negligable amount of time spent in
generateGid(). I am not sure what is causing it, but we definately don't need it in the MPI message, thus I remove it.