Repository navigation
Stride per-shard object checksum scan - #862
Open
amilnarski wants to merge 1 commit into
Open
amilnarski wants to merge 1 commit into
amilnarski wants to merge 1 commit into
Conversation
HollowObjectTypeReadStateShard.applyShardToChecksum walked every populated ordinal from 0 for each shard, making an object type's checksum O(S*M). Port the stride/round-up loop that 0333d1e applied to list, set and map shards so each shard visits only its own ordinals. Visit order is unchanged, so checksum values are identical; a new test pins the value for a sharded type with holes in the populated ordinals. Add a CheckSumObjects JMH benchmark alongside CheckSumCollections. At 100k records the old loop grows from 7.2ms (1 shard) to 62.6ms (128 shards); the new loop stays at ~7.2ms. At 40M records and 128 shards a checksum drops from 24.2s to 3.1s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HollowObjectTypeReadStateShard.applyShardToChecksum walked every populated ordinal from 0 for each shard, making an object type's checksum O(shards * ordinals). Port the stride/round-up loop that 0333d1e applied to list, set and map shards so each shard visits only its own ordinals. Visit order is unchanged, so checksum values are identical; a new test pins the value for a sharded type with holes in the populated ordinals. At 40M records and 128 shards a checksum drops from 24.2s to 3.1s.