Add Redis HNSW SQ8 support and SQ8 counterparts for every SVS benchmark

Makefile

@@ -247,6 +247,13 @@ integration-test-no-docker:
 	@echo "=== Running integration tests (assumes redis on port 6399) ==="
 	cargo test --test integration_redis --release -- --nocapture --test-threads=1
 
+# Run only against a dedicated Redis Search build with HNSW SQ8 support.
+# REDIS_TEST_PORT selects the instance; the test claims it before any FLUSHALL.
+.PHONY: integration-test-redis-sq8
+integration-test-redis-sq8:
+	@test -n "$(REDIS_TEST_PORT)" || (echo "Set REDIS_TEST_PORT to a dedicated HNSW SQ8 Redis instance"; exit 1)
+	cargo test --test integration_redis --release test_binary_redis_hnsw_sq8_reaches_server -- --ignored --exact --nocapture --test-threads=1
+
 # ============================================================
 # BENCHMARK - Run Rust microbenchmarks
 # ============================================================

datasets/datasets.json

@@ -1033,14 +1033,14 @@
     "description": "Knowledge embeddings"
   },
   {
-    "name": "dbpedia-openai-1M-1536-angular-100neighbors",
+    "name": "dbpedia-openai-975K-1536-angular-100neighbors",
     "vector_size": 1536,
     "distance": "cosine",
     "type": "tar",
-    "path": "dbpedia-openai-1M-1536-angular-100neighbors/dbpedia_openai_1M",
+    "path": "dbpedia-openai-975K-1536-angular-100neighbors/dbpedia_openai_975K",
     "link": "http://benchmarks.redislabs.s3.amazonaws.com/vecsim/laion400m/dbpedia_openai_1M.tgz",
-    "vector_count": 1000000,
-    "description": "Knowledge embeddings"
+    "vector_count": 975000,
+    "description": "Knowledge embeddings: full 975000-vector corpus with 100 neighbors per query; archive filename retains historical 1M label"
   },
   {
     "name": "h-and-m-2048-angular-filters",

docs/hnsw-sq8-e2e.md

@@ -0,0 +1,159 @@
+# HNSW SQ8 end-to-end benchmarks
+
+Use `experiments/configurations/redis-hnsw-sq8-k10.json` with a Redis Search build
+that supports HNSW `COMPRESSION SQ8`. The Rust Redis adapter reads compression
+and training threshold from `collection_params.hnsw_config`, sends them to
+`FT.CREATE`, and verifies the server's `FT.INFO` attributes before uploading.
+Unsupported servers fail explicitly.
+
+## Comparison
+
+The six configurations compare uncompressed HNSW, SQ8 with threshold 0, and SQ8
+with learned-mean threshold 10240, separately for FLOAT32 and FLOAT16 input.
+All use M=32, EF_CONSTRUCTION=200, upload parallelism 8 and batches of 64.
+The configurations explicitly request k=10, sweep ef=100/200/400/800/1600,
+and use 1/8/100 clients. These are starting settings, not claimed optimal settings.
+
+This follows the published SVS benchmark's comparison of index memory, total
+Redis memory, ingestion, throughput, latency and accuracy. It does not reproduce
+the original host or every original parameter:
+https://redis.io/blog/tech-dive-comprehensive-compression-leveraging-quantization-and-dimensionality-reduction/
+
+## Initial dataset
+
+Use `dbpedia-openai-100K-1536-angular`: the full registered 100K corpus and
+its own ground truth, with 10 neighbors per query. This PR provides the k=10
+starting matrix. Wider-k configurations and the remaining dimensionality/size
+matrix belong to follow-up benchmark work.
+
+Never use `--upload-end-idx` to shrink a corpus while retaining its full-corpus
+nearest-neighbor ground truth. Do not truncate or pad vectors to change dimension.
+For FLOAT16, report accuracy against the dataset's original ground truth as
+end-to-end quality including input conversion, and compare SQ8 with the FLOAT16
+baseline separately from FLOAT32.
+
+## Environment overrides
+
+Set `REDIS_HNSW_COMPRESSION` to `SQ8` and `REDIS_HNSW_TRAINING_THRESHOLD` to an
+integer from 0 to 102400. These variables apply only to HNSW configurations, not
+flat or SVS. Values in `collection_params.hnsw_config` (`COMPRESSION`,
+`TRAINING_THRESHOLD`) override the environment variables for each field.
+Whitespace is trimmed; an empty variable is treated as unset. An unparseable or
+out-of-range threshold value fails the run with an error naming the variable
+when the variable is actually used, that is when the configuration supplies no
+threshold; a config value wins and the variable is then not validated.
+Providing a threshold without compression from either source is an error.
+
+The resolved compression and threshold are recorded in result metadata as
+`hnsw_compression` and `hnsw_training_threshold`. The raw environment variables
+appear in the result's `env` record, only for HNSW configurations that build a
+vector index (the variables are not read otherwise).
+
+The `verify_ft_info` read-back runs only when the index is created. This guide's
+workflow is two invocations (`--skip-search` upload, then `--skip-upload`
+search). Export identical values for both invocations; otherwise the search
+results record `hnsw_compression` for an index that was built differently, and
+nothing on the search path detects it.
+
+The variables apply to every HNSW configuration that lacks `COMPRESSION`,
+including the `-plain` baselines in `redis-hnsw-sq8-k10.json` and
+`redis-svs-sq8-comparison.json`. Do not export them while running the paired
+matrices: a plain baseline would silently become SQ8 under its plain name, and
+a threshold alone makes every plain case fail. Check `hnsw_compression` and
+`hnsw_training_threshold` in the result metadata before comparing runs.
+
+## Execution
+
+Run on a dedicated Redis instance. Pin the harness commit, Redis and Search
+binaries (SHA256), datasets (SHA256), CPU, compiler and worker configuration.
+Use a fresh result directory per independent run. Keep only one configuration's
+index/data resident when measuring total Redis memory.
+
+Example first upload (from the repository root):
+
+```sh
+export REDIS_PORT=14960
+export REDIS_QUERY_TIMEOUT=90000
+./target/release/vector-db-benchmark \
+  --engines-file experiments/configurations/redis-hnsw-sq8-k10.json \
+  --engines redis-hnsw-float32-sq8-trained-k10 \
+  --datasets dbpedia-openai-100K-1536-angular \
+  --skip-search --keep-data --skip-if-exists false
+```
+
+Before search, inspect `FT.INFO idx:<config-name>` and
+`_FT.DEBUG VECSIM_INFO idx:<config-name> vector` on the dedicated server:
+
+- exact document count and no indexing errors;
+- requested compression and threshold;
+- `FRONTEND_INDEX.INDEX_SIZE=0`, `BACKEND_INDEX.INDEX_SIZE=dataset size`, and
+  `BACKGROUND_INDEXING=0` for the completed steady-state workload;
+- record `vector_index_sz_mb`, total `used_memory`, and the wait duration.
+
+The harness's existing `wait_for_indexing` checks document indexing, not the
+complete tiered vector migration. Its timeout also returns success. Therefore
+its upload time alone is not proof that the SQ8 backend finished building.
+Record upload plus the additional backend-drain duration as a separate quantity.
+Do not time dataset downloads as ingestion.
+
+Then run search on the validated corpus:
+
+```sh
+./target/release/vector-db-benchmark \
+  --engines-file experiments/configurations/redis-hnsw-sq8-k10.json \
+  --engines redis-hnsw-float32-sq8-trained-k10 \
+  --datasets dbpedia-openai-100K-1536-angular \
+  --skip-upload --keep-data --skip-if-exists false \
+  --fail-on-dropped-queries --repetitions 1 --search-duration 30
+```
+
+Set the dedicated server's `search-on-timeout` to `FAIL`. Drop the completed
+configuration's index with `FT.DROPINDEX idx:<config-name> DD` before the next
+configuration, after copying results and telemetry. Do not use `FLUSHALL` on a
+shared instance.
+
+## Regenerating configurations
+
+Run `python3 experiments/configurations/create-hnsw-sq8.py` to regenerate
+`redis-hnsw-sq8-k10.json` and `redis-svs-sq8-comparison.json`. Verify
+byte-for-byte identity with: `python3 experiments/configurations/create-hnsw-sq8.py
+&& git diff --exit-code experiments/configurations`.
+
+## Interpretation and regression checks
+
+Retain each independent run, not just a best-of result. Alternate baseline/SQ8
+order and run an identical-binary baseline A/A control before attributing small
+throughput differences. Repeat graph construction as well as searches.
+
+Use `mean_recall`, not `mean_precision_at_returned`, for recall@k. Require at
+least k valid ground-truth neighbors in every measured query. In particular,
+use k=10 for DBpedia 100K. Reject a measurement with dropped queries, partial corpus, schema mismatch, indexing
+errors or incomplete backend migration. Report both matching-parameter results
+and the best measured QPS meeting each recall target (for example 0.95 and 0.99);
+report an unreachable target rather than extrapolating a crossing.
+
+Compute vector-index compression as baseline `vector_index_sz_mb` divided by
+SQ8 `vector_index_sz_mb`. Report total Redis memory separately: stored HASH
+vectors remain full precision. Report ingestion through backend completion,
+QPS, p50/p95/p99 latency and recall@10 for each configuration.
+
+Performance regression thresholds must be calibrated from repeated controls;
+the first smoke run does not establish them. The separate transition workload
+must observe below-threshold accumulation, threshold crossing and backend drain
+while querying. A completed above-threshold index alone does not satisfy that
+transition measurement.
+
+## Validation
+
+Run `make agent-check`, then `REDIS_TEST_PORT=<dedicated-port> make integration-test-no-docker`
+and `REDIS_TEST_PORT=<dedicated-port> make integration-test-redis-sq8` on the
+selected build host. The integration suites destructively own the dedicated
+test instance; run them before loading benchmark data. The SQ8 test is a
+separate opt-in because the default Redis image may predate HNSW SQ8 support.
+It asserts server-reported compression and thresholds 0 and 4, and uploads
+enough vectors to cross the learned threshold.
+
+The existing downloaded-dataset shape test rejects DBpedia's empty filter
+objects (`conditions: {}`). Its independent fix is outside this PR. The standard
+clean-checkout gate does not exercise that downloaded corpus; do not report it
+as dataset coverage.

docs/svs-sq8-e2e.md

@@ -0,0 +1,142 @@
+# SVS and HNSW SQ8 end-to-end comparison
+
+Use `experiments/configurations/redis-svs-sq8-comparison.json` on a dedicated
+Redis instance with HNSW SQ8 support. These are client-to-Redis benchmarks.
+The [SQ8 execution guide](hnsw-sq8-e2e.md) covers binary provenance, schema
+readback, complete backend drain, cleanup, and repeated independent builds.
+
+## Paired workload
+
+Each input type (FLOAT32 and FLOAT16) has four cases:
+
+| Case suffix | Algorithm | Compression | HNSW training threshold |
+| --- | --- | --- | --- |
+| plain | HNSW | none | not applicable |
+| sq8-zero | HNSW | SQ8 | 0 |
+| sq8-trained | HNSW | SQ8 | 10240 |
+| svs-lvq8 | SVS-VAMANA | requested LVQ8 | not applicable |
+
+All cases use the same complete corpus, upload parallelism 8, batch size 64,
+k=10 and k=100, and 1/8/100 search clients. Each search setting sweeps
+100/200/400/800/1600 using HNSW `ef` or SVS `SEARCH_WINDOW_SIZE`, followed by
+calibration targets 0.95 and 0.99. This is 42 search points per index, 8 indexes.
+The existing k=10-only SQ8 file remains useful for datasets with narrow ground truth.
+
+HNSW uses M=32 and EF_CONSTRUCTION=200; SVS uses GRAPH_MAX_DEGREE=32 and
+CONSTRUCTION_WINDOW_SIZE=200. These are starting construction settings:
+equal numeric knobs do not imply equal graph structure or equal recall.
+Report each algorithm's measured speed/quality tradeoff.
+
+On the pinned Arm Neoverse-V2 development host build, requested SVS LVQ8 reports
+`GlobalSQ8` in FT.INFO and scalar quantization in VECSIM_INFO. Label results
+with that effective compression, CPU and build. They do not measure Intel LVQ
+performance. Read back the SVS graph degree, construction window, compression
+and actual training threshold; the HNSW threshold field does not configure SVS.
+
+## Dataset and workers
+
+Start with registered dataset `dbpedia-openai-975K-1536-angular-100neighbors`.
+This corrects the previous `dbpedia-openai-1M-1536-angular-100neighbors` entry:
+the archive contains 975,000 vectors, not 1,000,000. Its source URL is unchanged.
+Update saved commands to the corrected dataset name; existing downloads can be
+moved to `datasets/dbpedia-openai-975K-1536-angular-100neighbors/dbpedia_openai_975K`.
+
+Before measuring, verify the downloaded corpus count and that every measured
+query has at least 100 valid ground-truth neighbors. The registry name alone is
+not verification. The existing DBpedia 100K download has only 10 neighbors and
+must not be used with this combined k=10/k=100 matrix.
+
+For other SVS datasets, apply the same complete matrix to each corpus separately
+after checking its ground truth. Do not compare different embedding models as
+if only corpus size or dimension changed, shrink the corpus without recomputing
+ground truth, or truncate vectors.
+
+Run independent full builds with `search-workers=0` and `search-workers=4`,
+reading back the setting before each build. Worker count is a server setting;
+`upload_params.parallel` and search `parallel` are client counts. Repeat each
+build at least three times and alternate algorithm order. Keep only one case's
+index and keys resident when collecting memory measurements.
+
+## Run one case
+
+Build the harness with `make vector-db-benchmark` on the chosen build host.
+The harness writes to the checkout's `results/` directory. Archive it after each
+case, worker setting and repetition, then start the next case with an empty
+`results/` directory. Pin the harness commit, Redis/Search hashes, dataset
+hashes and effective settings.
+The example assumes the dedicated server already has the intended worker
+setting and `search-on-timeout=FAIL`.
+
+```sh
+export REDIS_PORT=14960
+export REDIS_QUERY_TIMEOUT=90000
+CASE=redis-compare-float32-sq8-trained
+DATASET=dbpedia-openai-975K-1536-angular-100neighbors
+CONFIG=experiments/configurations/redis-svs-sq8-comparison.json
+
+./target/release/vector-db-benchmark \
+  --engines-file "$CONFIG" --engines "$CASE" --datasets "$DATASET" \
+  --skip-search --keep-data --skip-if-exists false
+```
+
+Before searching, verify FT.INFO has the requested schema, complete document
+count and no indexing errors. Inspect `_FT.DEBUG VECSIM_INFO idx:$CASE vector`:
+frontend size must be 0, backend size must equal the corpus count and background
+indexing must be 0. Record upload plus additional backend-drain time separately
+from dataset download. The harness upload timer alone does not prove completion.
+
+```sh
+./target/release/vector-db-benchmark \
+  --engines-file "$CONFIG" --engines "$CASE" --datasets "$DATASET" \
+  --skip-upload --keep-data --skip-if-exists false \
+  --fail-on-dropped-queries --repetitions 1 --search-duration 30
+```
+
+Copy results and telemetry, then drop only this case's index and owned documents
+with `FT.DROPINDEX idx:$CASE DD` before the next case. Repeat for
+`redis-compare-float32-svs-lvq8`, both HNSW controls, and their FLOAT16 counterparts.
+
+## Compare results
+
+Report full build time through backend drain, vector-index memory, total Redis
+memory, QPS, p50/p95/p99 latency and `mean_recall` for each k and client count.
+FLOAT16 uses the original dataset ground truth, so its quality includes input
+conversion. Compare compression within each input type.
+
+Calibration uses the existing harness's `mean_precision_at_returned`, not recall,
+and searches breadth in [top, 1000]. Its `params.calibration.reached_target`
+flag is therefore not proof of recall at the requested target. Check the final
+measurement's `mean_recall`, full ground-truth width and absence of failed queries.
+Precision and recall agree only when all queries return k distinct results
+against k valid reference neighbors.
+
+For recall targets 0.95 and 0.99, select the best measured QPS that actually meets
+the target within the same dataset, dtype, k, worker count and client count.
+Include both fixed-sweep and calibrated points. The fixed sweep extends to 1600,
+beyond calibration's ceiling. Report a target as unachieved if no measured point
+qualifies; never infer a crossing or use precision as a substitute for recall.
+Retain all repetitions and use an identical-binary control before attributing
+small speed differences.
+
+This matrix measures complete ingestion and steady-state search. Threshold
+crossing and responsiveness during training are covered separately by the
+Redis transition runner, which lives on a separate follow-up branch
+(`dor-forer-MOD-14960-sq8-transition`).
+
+The `redis-svs-sq8-comparison.json` configuration is generated by
+`experiments/configurations/create-hnsw-sq8.py`.
+
+## SQ8 counterparts in the SVS calibration matrices
+
+Every SVS case in `dbpedia-calibration.json` and `cohere-calibration.json` has
+HNSW SQ8 counterparts (`*-cal-hnsw-sq8-zero-*` and `*-cal-hnsw-sq8-trained-*`,
+for training thresholds 0 and 10240) with the same data type, upload settings and
+search point (k=100, 100 clients, calibration to precision 0.95). The SVS matrix
+and the SQ8 matrix are therefore run with the same command, selecting either
+`--engines '<prefix>-cal-svs-*'` or `--engines '<prefix>-cal-hnsw-sq8-*'`, where
+`<prefix>` is `dbpedia` or `cohere`.
+
+The dbpedia file is meant for `dbpedia-openai-975K-1536-angular-100neighbors`,
+whose 100 neighbors per query are enough for k=100. The cohere file is meant for
+`cohere-768-1M`; the ground-truth width of that corpus must be verified to be at
+least 100 before reporting recall@100.