feat(format): track invalidated_fragments for index segments#5441
Draft
wjones127 wants to merge 6 commits intolance-format:mainfrom
Draft
feat(format): track invalidated_fragments for index segments#5441wjones127 wants to merge 6 commits intolance-format:mainfrom
invalidated_fragments for index segments#5441wjones127 wants to merge 6 commits intolance-format:mainfrom
Conversation
- Fix effective_fragment_bitmap() to subtract invalidated_fragments - Rename prune_updated_fields_from_indices to invalidate_updated_fields_in_indices - Add fragment IDs to invalidated_fragments instead of removing from fragment_bitmap - Add invalidation logic for DataReplacement operations - Fix TODOs in frag_reuse.rs and remapping.rs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
wjones127
changed the title
invalidated_fragments for index segments
wjones127
changed the title
invalidated_fragments for index segmentsinvalidated_fragments for index segments
- test_optimize_indices_resets_invalidated_fragments: verifies merged indices have empty invalidated_fragments - test_invalidated_fragments_lifecycle: verifies accumulation of invalidated fragments across multiple DataReplacement operations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For things like
DataReplacement, we need to start tracking within the index segments whether data in the index should be considered invalidated. This is slightly different than deleted, because the rows are still there in the same fragment, they just have been replaced in a new data file.This PR adds a field
invalidated_fragmentstoIndexMetadata, which is another fragment bitmap. The purpose offragment_bitmapis also clarified, particular that it should be considered immutable.Discussion here: #5453
Closes #5322, fixes #5321