feat: add LSM scanner to merge read MemWAL regions #5850
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This introduces an LSM (Log-Structured Merge) scanner that enables consistent reads across multiple data sources: - Base table (merged data, generation=0) - Flushed MemTables (persisted, generation=1,2,...) - Active MemTable (in-memory, highest generation) Key components: - LsmScanner: High-level API for LSM reads with deduplication - LsmDataSourceCollector: Collects data sources from base table and regions - LsmScanPlanner: Builds execution plan with Union + Dedup - DeduplicateExec: Deduplicates by PK, keeping highest generation - GenerationTagExec: Adds _gen and _rowaddr columns for dedup ordering Also includes: - mem_wal_read benchmark with DATASET_PREFIX support for S3 testing - active_memtable_ref() method on RegionWriter for LSM integration - Documentation fixes for generation numbering (unsigned, base=0) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
|
I have been thinking about this past 2 days, the part that we have to read each MemTable and then reverse the result feels just so inefficient to me. I think I have a good way to solve it now: When we scan MemTable, everything is in memory, reverse scan is fine. So when we flush MemTable, we should read the whole BatchStore in reverse order. This means the indexes also need to reverse the row position mapping, so the new row position is What do you think? |
Contributor
Author
|
Makes sense! Let me try update based on the current draft |
When flushing MemTable to disk, write data in reverse order (newest to oldest) so flushed generations are pre-sorted for K-way merge during LSM scan. This eliminates the need to reverse data during reads. Key changes: - BatchStore: add to_vec_reversed() that reverses batch order and rows - MemTable: add scan_batches_reversed() returning (batches, total_rows) - Flush: use reversed batches and pass total_rows to index creation - BTree index: add to_training_batches_reversed() with mapped positions - IVF-PQ index: add to_partition_batches_reversed() with mapped positions Row position mapping formula: flushed_pos = total_rows - original_pos - 1 Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When flushing MemTable to disk, write FTS index files directly from the in-memory FTS index without re-tokenizing the documents. This avoids duplicate tokenization work during flush. Key changes: - FtsMemIndex: add to_index_builder_reversed() that exports index data with reversed row positions for proper LSM ordering - InnerBuilder: add set_tokens/set_docs/set_posting_lists setters - InvertedIndexParams: add has_positions() getter - Flush: create_fts_indexes() now uses direct flush from in-memory data and properly commits index metadata to dataset manifest Row position mapping formula: flushed_pos = total_rows - original_pos - 1 Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change `to_vec_reversed()` to return `Result` instead of panicking on Arrow take kernel or RecordBatch creation errors - Replace `expect()` calls in `to_index_builder_reversed()` with proper `Error::io` returns for defensive error handling - Update callers to propagate errors appropriately Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on Jack's draft, separate out the scan reader part fist to be added without FTS and vector index. @jackye1995 can you take a look?