Skip to content

[MOD-19372] Fix flaky SVSTieredIndexTest.updateVectorsDuringUpdateJob - #1076

Open
nonirosenfeldredis wants to merge 1 commit into
mainfrom
MOD-19372-flaky-svs-update-vectors-test
Open

nonirosenfeldredis wants to merge 1 commit into
mainfrom
MOD-19372-flaky-svs-update-vectors-test

Conversation

@nonirosenfeldredis

@nonirosenfeldredis nonirosenfeldredis commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Jira: MOD-19372

Problem

SVSTieredIndexTest/1.updateVectorsDuringUpdateJob (uncompressed, multi-value) failed intermittently in CI (merge-queue run 37461178846) at the final per-label top-1 check:

tests/unit/test_svs_tiered.cpp:498: Failure
  id  Which is: 98
  i   Which is: 100

Root cause

Not data loss. Reproduced locally under heavy CPU load (about 6 failures in 1,100 runs with 24 parallel processes; 40/40 pass serially) and diagnosed the failing assertion: the label's own vector is stored in SVS at distance 0 from the query, the flat buffer is empty, and the same query with a window of 3000 returns the right label. The node is under-connected in the Vamana graph (same family as MOD-18994). How the mock workers batch the update-job drain depends on timing, so how under-connected a node ends up differs from run to run, and the test's search window of 250 occasionally did not reach it.

Fix

Test-only: remove the top-1 search at the end of the test. It checks graph reachability, which varies with timing and is not what this test covers. What remains still covers the update-during-update-job window: every label is held in exactly one tier, and the backend holds exactly the replacement vectors (indexSize() - numMarkedDeleted == n * new_per_label), which fails if a job carries a replaced vector into the backend after its delete.

The underlying graph-construction behavior (MOD-18994) is not changed here.

Test plan

  • SVSTieredIndexTest/*.updateVectorsDuringUpdateJob builds and passes (3 variants, ASan debug build).
  • clang-format 18.1.8 clean.
  • Not run locally: the rest of test_svs and the other tiered suites; relying on CI.

🤖 Generated with Claude Code


Note

Low Risk
Test-only change; no production code or indexing behavior is modified.

Overview
Test-only fix for intermittent CI failures in SVSTieredIndexTest.updateVectorsDuringUpdateJob (MOD-19372).

The PR drops the final block that ran top-1 KNN searches (with a widened search window of 250) on a few sample labels for uncompressed indexes. Those checks exercised approximate graph reachability after batch drains, which can vary with worker timing and under-connected Vamana nodes (same family as MOD-18994)—not the update-during-job correctness this test is meant to prove.

What still runs: updates while update jobs are in flight, then assertions that every label exists in exactly one tier and that indexSize() - getNumMarkedDeleted() equals n * new_per_label (the check that catches replaced vectors wrongly carried into the backend). No production or SVS graph behavior changes.

Reviewed by Cursor Bugbot for commit f2bed41. Bugbot is set up for automated code reviews on this repo. Configure here.

…ngUpdateJob

The final per-label top-1 search could miss a stored replacement vector under
heavy load: how the update job batches the flat buffer into the backend depends
on thread timing, so a node can end up under-connected in the Vamana graph and
the search window does not always reach it (MOD-18994 family). The vector is
stored correctly; only graph reachability varies, and that is graph quality,
not what this test is about.

The test already checks that every label is held in exactly one tier and that
the backend holds exactly the replacement vectors (live entries = n *
new_per_label), so remove the search.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.53%. Comparing base (ab6bb4f) to head (f2bed41).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1076      +/-   ##
==========================================
- Coverage   97.54%   97.53%   -0.02%     
==========================================
  Files         142      142              
  Lines        9405     9405              
==========================================
- Hits         9174     9173       -1     
- Misses        231      232       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants