Repository navigation
[MOD-19372] Fix flaky SVSTieredIndexTest.updateVectorsDuringUpdateJob - #1076
Open
nonirosenfeldredis wants to merge 1 commit into
Open
nonirosenfeldredis wants to merge 1 commit into
nonirosenfeldredis wants to merge 1 commit into
Conversation
…ngUpdateJob The final per-label top-1 search could miss a stored replacement vector under heavy load: how the update job batches the flat buffer into the backend depends on thread timing, so a node can end up under-connected in the Vamana graph and the search window does not always reach it (MOD-18994 family). The vector is stored correctly; only graph reachability varies, and that is graph quality, not what this test is about. The test already checks that every label is held in exactly one tier and that the backend holds exactly the replacement vectors (live entries = n * new_per_label), so remove the search. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1076 +/- ##
==========================================
- Coverage 97.54% 97.53% -0.02%
==========================================
Files 142 142
Lines 9405 9405
==========================================
- Hits 9174 9173 -1
- Misses 231 232 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
GuyAv46
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: MOD-19372
Problem
SVSTieredIndexTest/1.updateVectorsDuringUpdateJob(uncompressed, multi-value) failed intermittently in CI (merge-queue run 37461178846) at the final per-label top-1 check:Root cause
Not data loss. Reproduced locally under heavy CPU load (about 6 failures in 1,100 runs with 24 parallel processes; 40/40 pass serially) and diagnosed the failing assertion: the label's own vector is stored in SVS at distance 0 from the query, the flat buffer is empty, and the same query with a window of 3000 returns the right label. The node is under-connected in the Vamana graph (same family as MOD-18994). How the mock workers batch the update-job drain depends on timing, so how under-connected a node ends up differs from run to run, and the test's search window of 250 occasionally did not reach it.
Fix
Test-only: remove the top-1 search at the end of the test. It checks graph reachability, which varies with timing and is not what this test covers. What remains still covers the update-during-update-job window: every label is held in exactly one tier, and the backend holds exactly the replacement vectors (
indexSize() - numMarkedDeleted == n * new_per_label), which fails if a job carries a replaced vector into the backend after its delete.The underlying graph-construction behavior (MOD-18994) is not changed here.
Test plan
SVSTieredIndexTest/*.updateVectorsDuringUpdateJobbuilds and passes (3 variants, ASan debug build).clang-format18.1.8 clean.test_svsand the other tiered suites; relying on CI.🤖 Generated with Claude Code
Note
Low Risk
Test-only change; no production code or indexing behavior is modified.
Overview
Test-only fix for intermittent CI failures in
SVSTieredIndexTest.updateVectorsDuringUpdateJob(MOD-19372).The PR drops the final block that ran top-1 KNN searches (with a widened search window of 250) on a few sample labels for uncompressed indexes. Those checks exercised approximate graph reachability after batch drains, which can vary with worker timing and under-connected Vamana nodes (same family as MOD-18994)—not the update-during-job correctness this test is meant to prove.
What still runs: updates while update jobs are in flight, then assertions that every label exists in exactly one tier and that
indexSize() - getNumMarkedDeleted()equalsn * new_per_label(the check that catches replaced vectors wrongly carried into the backend). No production or SVS graph behavior changes.Reviewed by Cursor Bugbot for commit f2bed41. Bugbot is set up for automated code reviews on this repo. Configure here.