Skip to content

CRITICAL: random-100-match-kw-small-vocab declares vector_count 100 for a 1M corpus — shared-corpus mode uploads 100 docs and reports success #224

Closed

Description

@fcostaoliveira

Found by a 7-agent adversarial parity review; verified directly.

datasets/datasets.json:

random-100-match-kw-small-vocab-filters     vector_count=100        path=.../random_keywords_1m_vocab_10
random-100-match-kw-small-vocab-no-filters  vector_count=100        path=.../random_keywords_1m_vocab_10_no_filters
random-match-keyword-100-angular-filters    vector_count=1000000    path=.../random_keywords_1m
random-match-keyword-100-angular-no-filters vector_count=1000000    path=.../random_keywords_1m_no_filters

The two small-vocab entries point at a 1,000,000-point corpus (random_keywords_1m_vocab_10) while declaring vector_count: 100. Their siblings on the same corpus family declare 1000000 correctly. This looks like a copy-paste from the genuine random-100 entry directly above.

vector_count is not decorative. src/bin/vector_db_benchmark/engine/redis.rs:1901-1922 uses it as the shared-corpus completeness gate:

if let Some(expected) = dataset.config.vector_count.filter(|&n| n > 0) {

With shared-corpus mode enabled (redis.rs:249, driven by shared_key_prefix()), the upload is skipped as soon as existing >= expected — i.e. after 100 keys — and prints "Shared corpus already populated … skipping re-upload". The sweep then measures recall over ~100 documents against 1M-point ground truth. Recall collapses to ~0 with no error surfaced.

Fix: set both to 1000000. Then add a guard so this class cannot recur — e.g. validate vector_count against the actual corpus on first download, or fail loudly when the completeness gate is satisfied by an implausibly small fraction of a declared corpus.

Two more vector_count values are wrong in the benign (over-declaring) direction, measured from the on-disk NPY headers — worth fixing in the same PR since --list-datasets reports them:

  • arxiv-titles-384-angular-*: declared 2205995, actual (2138591, 384)
  • h-and-m-2048-angular-*: declared 105542, actual (105100, 2048)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions