Add Redis HNSW SQ8 support and k10 E2E configurations

Makefile

@@ -247,6 +247,13 @@ integration-test-no-docker:
 	@echo "=== Running integration tests (assumes redis on port 6399) ==="
 	cargo test --test integration_redis --release -- --nocapture --test-threads=1
 
+# Run only against a dedicated Redis Search build with HNSW SQ8 support.
+# REDIS_TEST_PORT selects the instance; the test claims it before any FLUSHALL.
+.PHONY: integration-test-redis-sq8
+integration-test-redis-sq8:
+	@test -n "$(REDIS_TEST_PORT)" || (echo "Set REDIS_TEST_PORT to a dedicated HNSW SQ8 Redis instance"; exit 1)
+	cargo test --test integration_redis --release test_binary_redis_hnsw_sq8_reaches_server -- --ignored --exact --nocapture --test-threads=1
+
 # ============================================================
 # BENCHMARK - Run Rust microbenchmarks
 # ============================================================

docs/hnsw-sq8-e2e.md

@@ -0,0 +1,122 @@
+# HNSW SQ8 end-to-end benchmarks
+
+Use `experiments/configurations/redis-hnsw-sq8-k10.json` with a Redis Search build
+that supports HNSW `COMPRESSION SQ8`. The Rust Redis adapter reads compression
+and training threshold from `collection_params.hnsw_config`, sends them to
+`FT.CREATE`, and verifies the server's `FT.INFO` attributes before uploading.
+Unsupported servers fail explicitly.
+
+## Comparison
+
+The six configurations compare uncompressed HNSW, SQ8 with threshold 0, and SQ8
+with learned-mean threshold 10240, separately for FLOAT32 and FLOAT16 input.
+All use M=32, EF_CONSTRUCTION=200, upload parallelism 8 and batches of 64.
+The configurations explicitly request k=10, sweep ef=100/200/400/800/1600,
+and use 1/8/100 clients. These are starting settings, not claimed optimal settings.
+
+This follows the published SVS benchmark's comparison of index memory, total
+Redis memory, ingestion, throughput, latency and accuracy. It does not reproduce
+the original host or every original parameter:
+https://redis.io/blog/tech-dive-comprehensive-compression-leveraging-quantization-and-dimensionality-reduction/
+
+## Initial dataset
+
+Use `dbpedia-openai-100K-1536-angular`: the full registered 100K corpus and
+its own ground truth, with 10 neighbors per query. This PR provides the k=10
+starting matrix. Wider-k configurations and the remaining dimensionality/size
+matrix belong to follow-up benchmark work.
+
+Never use `--upload-end-idx` to shrink a corpus while retaining its full-corpus
+nearest-neighbor ground truth. Do not truncate or pad vectors to change dimension.
+For FLOAT16, report accuracy against the dataset's original ground truth as
+end-to-end quality including input conversion, and compare SQ8 with the FLOAT16
+baseline separately from FLOAT32.
+
+## Execution
+
+Run on a dedicated Redis instance. Pin the harness commit, Redis and Search
+binaries (SHA256), datasets (SHA256), CPU, compiler and worker configuration.
+Use a fresh result directory per independent run. Keep only one configuration's
+index/data resident when measuring total Redis memory.
+
+Example first upload (from the repository root):
+
+```sh
+export REDIS_PORT=14960
+export REDIS_QUERY_TIMEOUT=90000
+./target/release/vector-db-benchmark \
+  --engines-file experiments/configurations/redis-hnsw-sq8-k10.json \
+  --engines redis-hnsw-float32-sq8-trained-k10 \
+  --datasets dbpedia-openai-100K-1536-angular \
+  --skip-search --keep-data --skip-if-exists false
+```
+
+Before search, inspect `FT.INFO idx:<config-name>` and
+`_FT.DEBUG VECSIM_INFO idx:<config-name> vector` on the dedicated server:
+
+- exact document count and no indexing errors;
+- requested compression and threshold;
+- `FRONTEND_INDEX.INDEX_SIZE=0`, `BACKEND_INDEX.INDEX_SIZE=dataset size`, and
+  `BACKGROUND_INDEXING=0` for the completed steady-state workload;
+- record `vector_index_sz_mb`, total `used_memory`, and the wait duration.
+
+The harness's existing `wait_for_indexing` checks document indexing, not the
+complete tiered vector migration. Its timeout also returns success. Therefore
+its upload time alone is not proof that the SQ8 backend finished building.
+Record upload plus the additional backend-drain duration as a separate quantity.
+Do not time dataset downloads as ingestion.
+
+Then run search on the validated corpus:
+
+```sh
+./target/release/vector-db-benchmark \
+  --engines-file experiments/configurations/redis-hnsw-sq8-k10.json \
+  --engines redis-hnsw-float32-sq8-trained-k10 \
+  --datasets dbpedia-openai-100K-1536-angular \
+  --skip-upload --keep-data --skip-if-exists false \
+  --fail-on-dropped-queries --repetitions 1 --search-duration 30
+```
+
+Set the dedicated server's `search-on-timeout` to `FAIL`. Drop the completed
+configuration's index with `FT.DROPINDEX idx:<config-name> DD` before the next
+configuration, after copying results and telemetry. Do not use `FLUSHALL` on a
+shared instance.
+
+## Interpretation and regression checks
+
+Retain each independent run, not just a best-of result. Alternate baseline/SQ8
+order and run an identical-binary baseline A/A control before attributing small
+throughput differences. Repeat graph construction as well as searches.
+
+Use `mean_recall`, not `mean_precision_at_returned`, for recall@k. Require at
+least k valid ground-truth neighbors in every measured query. In particular,
+use k=10 for DBpedia 100K. Reject a measurement with dropped queries, partial corpus, schema mismatch, indexing
+errors or incomplete backend migration. Report both matching-parameter results
+and the best measured QPS meeting each recall target (for example 0.95 and 0.99);
+report an unreachable target rather than extrapolating a crossing.
+
+Compute vector-index compression as baseline `vector_index_sz_mb` divided by
+SQ8 `vector_index_sz_mb`. Report total Redis memory separately: stored HASH
+vectors remain full precision. Report ingestion through backend completion,
+QPS, p50/p95/p99 latency and recall@10 for each configuration.
+
+Performance regression thresholds must be calibrated from repeated controls;
+the first smoke run does not establish them. The separate transition workload
+must observe below-threshold accumulation, threshold crossing and backend drain
+while querying. A completed above-threshold index alone does not satisfy that
+transition measurement.
+
+## Validation
+
+Run `make agent-check`, then `REDIS_TEST_PORT=<dedicated-port> make integration-test-no-docker`
+and `REDIS_TEST_PORT=<dedicated-port> make integration-test-redis-sq8` on the
+selected build host. The integration suites destructively own the dedicated
+test instance; run them before loading benchmark data. The SQ8 test is a
+separate opt-in because the default Redis image may predate HNSW SQ8 support.
+It asserts server-reported compression and thresholds 0 and 4, and uploads
+enough vectors to cross the learned threshold.
+
+The existing downloaded-dataset shape test rejects DBpedia's empty filter
+objects (`conditions: {}`). Its independent fix is outside this PR. The standard
+clean-checkout gate does not exercise that downloaded corpus; do not report it
+as dataset coverage.

experiments/configurations/redis-hnsw-sq8-k10.json

@@ -0,0 +1,140 @@
+[
+  {
+    "name": "redis-hnsw-float32-plain-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT32", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT32"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  },
+  {
+    "name": "redis-hnsw-float32-sq8-zero-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT32", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200, "COMPRESSION": "SQ8", "TRAINING_THRESHOLD": 0}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT32"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  },
+  {
+    "name": "redis-hnsw-float32-sq8-trained-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT32", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200, "COMPRESSION": "SQ8", "TRAINING_THRESHOLD": 10240}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT32"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  },
+  {
+    "name": "redis-hnsw-float16-plain-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT16", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT16"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  },
+  {
+    "name": "redis-hnsw-float16-sq8-zero-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT16", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200, "COMPRESSION": "SQ8", "TRAINING_THRESHOLD": 0}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT16"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  },
+  {
+    "name": "redis-hnsw-float16-sq8-trained-k10",
+    "engine": "redis",
+    "collection_params": {"algorithm": "hnsw", "data_type": "FLOAT16", "hnsw_config": {"M": 32, "EF_CONSTRUCTION": 200, "COMPRESSION": "SQ8", "TRAINING_THRESHOLD": 10240}},
+    "upload_params": {"parallel": 8, "batch_size": 64, "data_type": "FLOAT16"},
+    "search_params": [
+      {"parallel": 1, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 1, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 8, "top": 10, "search_params": {"ef": 1600}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 100}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 200}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 400}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 800}},
+      {"parallel": 100, "top": 10, "search_params": {"ef": 1600}}
+    ]
+  }
+]

src/bin/vector_db_benchmark/config.rs

@@ -34,6 +34,22 @@ pub struct HnswConfig {
         alias = "ef_construction"
     )]
     pub ef_construction: Option<i64>,
+    /// Redis HNSW scalar compression. Serde rejects malformed and explicit null
+    /// values; the library validates SQ8 support against algorithm and type.
+    #[serde(
+        rename = "COMPRESSION",
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "vector_db_benchmark::redis_hnsw_sq8::present_string"
+    )]
+    pub compression: Option<String>,
+    #[serde(
+        rename = "TRAINING_THRESHOLD",
+        default,
+        skip_serializing_if = "Option::is_none",
+        deserialize_with = "vector_db_benchmark::redis_hnsw_sq8::present_u64"
+    )]
+    pub training_threshold: Option<u64>,
     /// Qdrant: keep the HNSW graph on disk (mmap) instead of in RAM.
     pub on_disk: Option<bool>,
     /// Qdrant: per-payload-value graph links. `m: 0` + `payload_m: k` builds
@@ -862,6 +878,35 @@ mod tests {
     use super::*;
     use serde_json::json;
 
+    #[test]
+    fn redis_sq8_fields_are_typed_and_reject_malformed_values() {
+        let cfg: HnswConfig = serde_json::from_value(json!({
+            "M": 16, "COMPRESSION": "sq8", "TRAINING_THRESHOLD": 0
+        }))
+        .unwrap();
+        assert_eq!(cfg.compression.as_deref(), Some("sq8"));
+        assert_eq!(cfg.training_threshold, Some(0));
+        assert!(cfg.unsupported_keys().is_empty());
+        let serialized = serde_json::to_value(&cfg).unwrap();
+        assert_eq!(serialized["COMPRESSION"], "sq8");
+        assert_eq!(serialized["TRAINING_THRESHOLD"], 0);
+        let absent = serde_json::to_value(HnswConfig::default()).unwrap();
+        assert!(absent.get("COMPRESSION").is_none());
+        assert!(absent.get("TRAINING_THRESHOLD").is_none());
+        serde_json::from_value::<HnswConfig>(absent).unwrap();
+
+        for bad in [
+            json!({"COMPRESSION": null}),
+            json!({"COMPRESSION": 8}),
+            json!({"TRAINING_THRESHOLD": null}),
+            json!({"TRAINING_THRESHOLD": -1}),
+            json!({"TRAINING_THRESHOLD": "0"}),
+            json!({"TRAINING_THRESHOLD": 1.5}),
+        ] {
+            assert!(serde_json::from_value::<HnswConfig>(bad).is_err());
+        }
+    }
+
     /// Corpus size advertised by a dataset's own corpus filename, when the leaf
     /// path segment carries an unambiguous magnitude token — `random_keywords_1m`
     /// => 1_000_000, `..._100k` => 100_000, `...-1G-...` => 1_000_000_000. The
@@ -2016,6 +2061,8 @@ mod shipped_config_knob_guard {
             &[
                 "m",
                 "ef_construction",
+                "compression",
+                "training_threshold",
                 "on_disk",
                 "payload_m",
                 "inline_storage",
@@ -2059,6 +2106,8 @@ mod shipped_config_knob_guard {
         match leaf {
             "M" => "m",
             "EF_CONSTRUCTION" | "ef_construct" => "ef_construction",
+            "COMPRESSION" => "compression",
+            "TRAINING_THRESHOLD" => "training_threshold",
             // #215 made `EF` an alias of the typed `ef` field.
             "EF" => "ef",
             other => other,
@@ -2590,6 +2639,8 @@ mod shipped_config_knob_guard {
             "M",
             "EF_CONSTRUCTION",
             "ef_construct",
+            "COMPRESSION",
+            "TRAINING_THRESHOLD",
             "on_disk",
             "payload_m",
             "inline_storage",