[ENH] Prefetch block by prefixes

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • N/A
• New functionality
  • Implements blockfile prefetch by prefix

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[Sicheng-Pan]

• [ENH] Optimize regex algorithm #4624
• [ENH] Prefetch block by prefixes #4623 👈 (View in Graphite)
• [TST] Regex testing #4570
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

[propel-code-bot]

Implement Blockfile Prefetch by Prefix

This PR introduces the ability to prefetch blockfile data based on a list of prefixes, expanding efficient bulk loading strategies for the blockstore. The core addition is a new method on the sparse index (get_block_ids_for_prefixes) that retrieves all relevant block IDs for given prefixes, and this is now threaded through the Arrow blockfile implementation and exposed via the blockfile reader API.

Key Changes:
• Added get_block_ids_for_prefixes method to SparseIndexReader and SparseIndexWriter to enable efficient lookup of block IDs covering multiple prefixes.
• Introduced load_blocks_for_prefixes async method in both ArrowBlockfileReader and the generic BlockfileReader interface, which prefetches all blocks covering the given prefixes.
• Hooked up these methods in the blockfile reader trait hierarchy to allow client code to trigger block prefetch using a list of prefixes.
• Small refactor to get_block_ids_range signature to remove an unused lifetime.
• Maintained unimplemented!() for the memory-backed reader variant with a placeholder for future support.

Affected Areas:
• arrow/sparse_index.rs (SparseIndex lookup logic)
• arrow/blockfile.rs (prefetch logic)
• types/reader.rs (API exposure and trait implementations)

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[Documentation]

The new get_block_ids_for_prefixes function could benefit from some documentation comments explaining its purpose, behavior, and usage - especially since it's a key component of the newly added functionality.

[Sicheng-Pan]

Merge activity

• May 29, 11:02 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• May 29, 11:04 PM UTC: Graphite rebased this pull request as part of a merge.
• May 29, 11:37 PM UTC: Graphite couldn't merge this PR because it was not satisfying all requirements (Failed CI: 'all-required-pr-checks-passed', 'Python tests / test-cluster-rust-frontend (3.9, blacksmith-16vcpu-ubuntu-2204, chromadb/test/api)', 'Python tests / test-cluster-rust-frontend (3.9, blacksmith-16vcpu-ubuntu-2204, chromadb/test/property/test_colle...', 'Rust tests / test-integration (blacksmith-16vcpu-ubuntu-2204)').
• May 30, 6:07 PM UTC: @Sicheng-Pan merged this pull request with Graphite.