Skip to content

fix!: bump IVF_RQ version for compatibility check#6097

Merged
wjones127 merged 4 commits intomainfrom
yang/add-ivfrq-compatibility-checks
Mar 4, 2026
Merged

fix!: bump IVF_RQ version for compatibility check#6097
wjones127 merged 4 commits intomainfrom
yang/add-ivfrq-compatibility-checks

Conversation

@BubbleCal
Copy link
Contributor

@BubbleCal BubbleCal commented Mar 4, 2026

Summary

  • add IVF_RQ compatibility tests that run against the latest stable pylance release to ensure read/query compatibility both directions
  • update IndexDetails to report the actual highest vector version rather than a single constant

@github-actions github-actions bot added the python label Mar 4, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@BubbleCal BubbleCal changed the title Track vector index versions per type fix: bump IVF_RQ version for compatibility check Mar 4, 2026
@github-actions github-actions bot added the bug Something isn't working label Mar 4, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Review

P1: index_version derived from all old_indices, not just the merged subset

In rust/lance/src/index/append.rs, the new code takes old_indices.iter().map(|idx| idx.index_version).max() over all old indices, but only the last indices_merged entries are being merged. The fragment bitmap correctly scopes to merged_indices, but the version derivation doesn't. While in practice all entries in old_indices should be the same type/version (validated at line 102), using merged_indices.iter() instead of old_indices.iter() would be more precise and consistent with the bitmap logic:

let index_version = merged_indices
    .iter()
    .map(|idx| idx.index_version)
    .max()
    .ok_or_else(|| ...)?;

P1: max_vector_version() manually enumerates all vector types

IndexType::max_vector_version() lists all vector index variants by hand. If a new vector index type is added without updating this list, the max version will be silently wrong. Consider either:

  • Deriving the list from IndexType variants (e.g., using strum or a similar enum iterator), or
  • Adding a compile-time or test assertion that covers all IndexType variants.

Nit: Compat test missing check_write

IvfRqVectorIndex only has check_read — no check_write (unlike the IvfPqVectorIndex test above it). If the intent is to also verify write-path compatibility (append + optimize on an index created by a different version), consider adding it.

@BubbleCal BubbleCal requested a review from westonpace March 4, 2026 16:01
@BubbleCal BubbleCal changed the title fix: bump IVF_RQ version for compatibility check fix!: bump IVF_RQ version for compatibility check Mar 4, 2026
@Xuanwo
Copy link
Collaborator

Xuanwo commented Mar 4, 2026

Hi @BubbleCal, I think the review raised by claude is valid:

index_version derived from all old_indices, not just the merged subset

Do we need to take care about this?

@codecov
Copy link

codecov bot commented Mar 4, 2026

Codecov Report

❌ Patch coverage is 91.42857% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/index.rs 25.00% 2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@BubbleCal
Copy link
Contributor Author

Hi @BubbleCal, I think the review raised by claude is valid:

index_version derived from all old_indices, not just the merged subset

Do we need to take care about this?

it's common that we just append a delta index and merged_num=0 in that case, so we need to derive index_version from all old_indices.

I think today all old_indices will always be with the same version, even on a new version of lance, we always respect the existing index's version when incremental indexing

Copy link
Collaborator

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR!

When appending to an existing index, the new delta's index_version was
set to max(old_versions), but indices created by old Lance wrote
index_version=0 (unset). This caused the delta to inherit version 0,
making old Lance fall back to brute-force KNN instead of using the index.

Fix by taking max(old_versions, current_library_version_for_type), so the
appended index always declares at least the minimum required version for
its index type.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@wjones127 wjones127 force-pushed the yang/add-ivfrq-compatibility-checks branch from c691663 to 280e9a5 Compare March 4, 2026 20:29
Copy link
Contributor

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay this should go good to go. Will merge once CI is green and we can make a release.

@wjones127 wjones127 merged commit 6f48c26 into main Mar 4, 2026
29 checks passed
@wjones127 wjones127 deleted the yang/add-ivfrq-compatibility-checks branch March 4, 2026 21:22
wjones127 added a commit to wjones127/lance that referenced this pull request Mar 4, 2026
Summary
- add IVF_RQ compatibility tests that run against the latest stable
pylance release to ensure read/query compatibility both directions
- update IndexDetails to report the actual highest vector version rather
than a single constant

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change bug Something isn't working python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants