Add optional MurmurHash support #142
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kusalk
force-pushed
the
murmur-hash-alg
branch
5 times, most recently
from
1a11e95 to
a5747d2
Compare
kusalk
force-pushed
the
murmur-hash-alg
branch
from
a5747d2 to
d15a68c
Compare
kusalk
force-pushed
the
murmur-hash-alg
branch
from
d15a68c to
bad9b9a
Compare
kusalk
commented
Feb 17, 2025
| private static final String SELECTED_ALGORITHM_PROP = "hash.algorithm"; | ||
| private static final HashAlgorithm DEFAULT_ALGORITHM = HashAlgorithm.MD5; | ||
| private static final HashAlgorithm SELECTED_ALGORITHM = | ||
| HashAlgorithm.valueOf(System.getProperty(SELECTED_ALGORITHM_PROP, DEFAULT_ALGORITHM.name()).toUpperCase()); |
There was a problem hiding this comment.
We read the chosen hashing algorithm from the system property here
kusalk
changed the title
Scavenger Test Results166 files 166 suites 1m 31s ⏱️ Results for commit 7554c09. ♻️ This comment has been updated with latest results. |
kusalk
force-pushed
the
murmur-hash-alg
branch
from
b804064 to
f465a85
Compare
kusalk
commented
Feb 22, 2025
| private static String from(String signature) { | ||
| MessageDigest md = callWithCheckedExceptionWrapping(() -> MessageDigest.getInstance("MD5")); | ||
| md.update(signature.getBytes(StandardCharsets.UTF_8)); | ||
| // Ideally should be "%032x" to avoid confusion from leading zeros being dropped, do not change now, avoid compat issues |
There was a problem hiding this comment.
I noticed this peculiarity whilst testing - the returned String won't always be 32 characters. This shouldn't cause issues but noted it in a comment. We obviously can't change it now anyway as it would cause backwards compatibility issues.
kusalk
force-pushed
the
murmur-hash-alg
branch
from
f465a85 to
7554c09
Compare
kusalk
commented
Feb 22, 2025
| @@ -17,12 +22,19 @@ dependencies { | |||
| implementation("io.grpc:grpc-kotlin-stub:${property("grpcKotlinVersion")}") | |||
| implementation("io.grpc:grpc-protobuf:${property("grpcVersion")}") | |||
| implementation("javax.annotation:javax.annotation-api:1.3.2") | |||
| implementation("commons-codec:commons-codec:1.18.0") | |||
There was a problem hiding this comment.
We use the MurmurHash implementation from the commons-codec library dependency - it will increase the size of the agent JAR slightly (~300 kB)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The changes encompassed in this PR were originally included with those in #134. I extracted these change into this separate PR following a discussion in the former PR.
This PR introduces support for MurmurHash as an alternative to the default MD5. In my experience it is over 4x faster at hashing (rough benchmarks below).
MurmurHash can be activated by providing the following argument to the agent
-Dhash.algorithm=murmur. The argument must be supplied to the collector instead if using the legacy agent.Benchmark:
MethodRegistry#getHashScenario:
ConcurrentHashMapcache disabled