Add Redis HNSW SQ8 support and SQ8 counterparts for every SVS benchmark

[dor-forer]

E2E part of MOD-14960: make HNSW SQ8 benchmarkable through this harness and give every SVS-VAMANA benchmark in the repository an SQ8 counterpart. Separate from the VectorSimilarity microbenchmarks (#1064, #1065, #1067). Replaces #324 and the fork draft #323.

1. Redis engine: HNSW SQ8 support

The Redis adapter ignored HNSW compression settings, so a configuration labeled SQ8 created an uncompressed index. It now reads collection_params.hnsw_config.COMPRESSION / TRAINING_THRESHOLD (typed and validated: SQ8 only, threshold 0..=102400, FLOAT32/FLOAT16, HNSW only), emits them in FT.CREATE, verifies the created schema through FT.INFO, and records declared and effective values in result metadata. A dedicated make integration-test-redis-sq8 target runs against SQ8-capable servers (opt-in, because the CI Redis image predates HNSW SQ8). Run guide: docs/hnsw-sq8-e2e.md.

2. SQ8 counterparts for every SVS benchmark

The SVS E2E benchmarks are the two calibration matrices dbpedia-calibration.json and cohere-calibration.json. Every SVS case there now has HNSW SQ8 counterparts, <prefix>-cal-hnsw-sq8-zero-{float16,float32} (threshold 0) and <prefix>-cal-hnsw-sq8-trained-{float16,float32} (threshold 10240), identical to the file's HNSW baselines except for the compression keys: same data type, upload settings and search point (k=100, 100 clients, calibration to 0.95). Pure additions. The SVS matrix and the SQ8 matrix run with the same command, selecting --engines '<prefix>-cal-svs-*' or --engines '<prefix>-cal-hnsw-sq8-*'.

3. Additional configurations

• redis-hnsw-sq8-k10.json: FLOAT32/FLOAT16 x plain HNSW, SQ8 threshold 0, SQ8 threshold 10240; k=10 for the DBpedia 100K corpus, whose ground truth has only ten neighbors per query.
• redis-svs-sq8-comparison.json: the same four index types plus SVS-VAMANA, k=10 and k=100, 1/8/100 clients, breadth sweep and calibration 0.95/0.99, for the full DBpedia 975K corpus. Guide: docs/svs-sq8-e2e.md.
• create-hnsw-sq8.py regenerates both files byte-identically (python3 experiments/configurations/create-hnsw-sq8.py && git diff --exit-code experiments/configurations).

4. Environment overrides

REDIS_HNSW_COMPRESSION and REDIS_HNSW_TRAINING_THRESHOLD, mirroring REDIS_SVS_COMPRESSION / REDIS_SVS_REDUCE: a value in hnsw_config wins per field; the variables are read only for HNSW configurations that build a vector index; env text is trimmed and an empty value is unset. Unlike the SVS variables, an unparseable or out-of-range threshold is a fatal error naming the variable. Both are registered in the KNOWN_UNRECORDED inventory. The docs warn that the values must be identical for the upload and the search invocations (read-back runs only at index creation) and that the variables also convert the -plain baselines, so they must not be exported while running the shipped matrices.

5. Dataset registry correction (breaking rename)

dbpedia-openai-1M-1536-angular-100neighbors declared 1,000,000 vectors, but the archive holds 975,000 (verified on disk; the harness only warned). Renamed to dbpedia-openai-975K-1536-angular-100neighbors with vector_count 975000 and matching path components; source URL unchanged. Move existing downloads to datasets/dbpedia-openai-975K-1536-angular-100neighbors/dbpedia_openai_975K/ to avoid a re-download. The legacy v0/ registry is untouched.

Validation

• make agent-check (fmt, clippy, unit tests, harness invariants): passed at every commit; 185 unit tests at HEAD (x86, rustc 1.98.1; also Arm for the engine change).
• make integration-test-no-docker against redis:8.8.0: 44 passed, 1 ignored (the SQ8 test).
• make integration-test-redis-sq8 against the SQ8-capable Search development build (Arm): passed, including the env-override case (config thresholds 0 and 4 kept, env threshold 8 applied to the config without COMPRESSION). Mutation check on the read-back: forcing an emitted threshold of 0 made the learned-threshold case fail with differs from requested 4.
• Negative control: a run with REDIS_HNSW_TRAINING_THRESHOLD=abc exited 1 with REDIS_HNSW_TRAINING_THRESHOLD must be an integer from 0 to 102400, got "abc" before creating any index.
• Generator round-trip: git diff --exit-code experiments/configurations clean after regeneration.
• Each commit had a separate spec-and-quality review and the branch a whole-branch review.

GitHub CI here runs code checks and the general integration matrix; it does not run the Arm benchmarks. Integration tests (Milvus) fails at the "Start Milvus" step on this branch and on master (2026-09-24); unrelated to this change.

Initial Arm results (smoke, not throughput evidence)

DBpedia 100K, 1536d, cosine, first 100 queries, k=10, ef=400, one client, one repetition, four Search workers. Redis 8.10.1 with the Search development build at 8d1dae1 (HNSW SQ8 targets Redis 8.12).

Input	Index	Vector index MiB	Total Redis used memory MiB	Recall@10
FLOAT32	Uncompressed	629.96	1461.88	99.8%
FLOAT32	SQ8, threshold 0	190.12	949.55	96.6%
FLOAT32	SQ8, threshold 10240	190.58	949.75	99.0%
FLOAT16	Uncompressed	336.01	777.13	97.9%
FLOAT16	SQ8, threshold 0	190.12	607.76	97.1%
FLOAT16	SQ8, threshold 10240	190.58	607.95	99.2%

Every case verified the requested schema, full document and backend counts, empty frontend and no indexing errors before searching; one index resident at a time. Total memory includes the stored HASH vectors. Full-corpus 975K runs and the training-transition study are reported on MOD-14960.

Notes and follow-ups

• Unit tests that construct RedisEngine on HNSW configs read the ambient environment, as they already do for REDIS_SVS_*.
• Follow-up for both REDIS_HNSW_* and REDIS_SVS_*: run the FT.INFO read-back on the --skip-upload search path too.
• The downloaded-dataset shape test rejects DBpedia's empty filter objects (conditions: {}); that independent test fix is not part of this PR.
• Remaining MOD-14960 work outside this PR: the 64/128-dimension datasets, repeated runs for regression thresholds, and refreshing the Search build to current master before publishing numbers.

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

I read this against the base tree. It's stacked on #324, so anything that lives only in #324 (the hnsw_compression / hnsw_training_threshold metadata recording, redis-hnsw-sq8-k10.json, verify_ft_info) doesn't exist in base and I couldn't check it here. The author already covers most of what I'd have raised in the PR body: the upload/search env mismatch, -plain baselines being converted, and ambient env leaking into unit tests. So below I only list what's left.

Worth fixing before merge

Nothing blocking in this diff. The precedence logic is in src/redis_hnsw_sq8.rs (Sq8Options::resolve), not in src/bin/, so it's unit-tested. The tests pin each part of the contract separately: config wins per field, config 0 beats env 4, blank env counts as unset, a threshold without compression is an error, the range edge at 102400/999999, and unparseable env input errors and names the variable. Treating a bad threshold as fatal is stricter than REDIS_SVS_REDUCE, which quietly drops a value that won't parse. That's the right choice here.

Optional / follow-up

1. Blast radius is wider than the SVS equivalent. REDIS_SVS_COMPRESSION can only reach configs whose algorithm resolves to SVS. The new gate at engine/redis.rs:~225 is algorithm == "hnsw" && !skip_vector_index, and hnsw is the default when no algorithm is set (redis.rs:208). So an exported REDIS_HNSW_COMPRESSION=SQ8 turns every Redis HNSW config without COMPRESSION into SQ8 under its original name. That includes the plain calibration baselines and presumably most of redis-single-node.json. This is the repo's usual failure mode: the run still completes and still reports a number, under a wrong label. The docs and the recorded hnsw_compression catch it only if someone reads the metadata. A cheap guard is a one-line stderr notice in RedisEngine::new whenever compression or threshold came from the env rather than config (resolve already computes compression_from_env). Not a blocker, since it matches how the SVS vars behave.
2. The "never touches flat/SVS" gate has no test that runs in CI. The read_sq8_env condition lives in src/bin/ and isn't unit-tested. The only end-to-end check is test_binary_redis_hnsw_sq8_reaches_server, which is #[ignore] and needs an SQ8-capable Search build, so CI never runs it. If the gate regressed to reading env for every algorithm, the tests CI runs would stay green. Moving the algorithm/skip_vector_index gating into resolve (or a small pure helper in src/redis_hnsw_sq8.rs) would let a unit test pin it. Worth checking that the test fails when the gate is deleted.
3. FT.INFO read-back doesn't run on the --skip-upload path. This is already listed as a follow-up, and it's the more substantive of the two gaps. Right now hnsw_compression on a search-only run records what the env said, not what the index was built with. Worth filing as its own issue (for both REDIS_HNSW_* and REDIS_SVS_*) rather than leaving it in PR prose.
4. Dataset rename (datasets/datasets.json:1036). This looks correct. extract_tgz unpacks into datasets/<path>/ (download.rs:202-213), so changing the inner dir to dbpedia_openai_975K is harmless for a fresh download. Nothing outside v0/ references the old name. Two side effects to be aware of:
   • Results filed under the old name no longer group with new ones.
   • The fix corrects the declared count but only produces a warning today. The corpus-identity check still trusts vector_count rather than verifying the data (taxonomy item 3), which is out of scope here.
5. Generator drift. create-hnsw-sq8.py is only checked by the manual git diff --exit-code round-trip. Nothing in CI stops someone from editing a generated JSON by hand. That's consistent with the other create-*.py scripts, so it's fine to leave.
6. Docs. docs/svs-sq8-e2e.md points at an unmerged branch (dor-forer-MOD-14960-sq8-transition) and an internal ticket ID. It would be better to point at a PR or issue once one exists, the same way the personal fork link was already dropped in F5.

On the matrices themselves: the new *-cal-hnsw-sq8-* entries in dbpedia-calibration.json and cohere-calibration.json copy each file's HNSW baseline field for field (dbpedia M=16/L2, cohere M=32/IP, same upload/search point) and add only the two compression keys. That's the right comparison: SQ8 against plain HNSW, not against SVS's GRAPH_MAX_DEGREE=32.

Net: clean, carefully self-reviewed, and well-scoped for a first contribution. The remaining risk is mislabeled results when env vars are exported and nobody checks the metadata, and items 1–3 would narrow it. Whether to accept, and whether to wait for #324 first, is the maintainer's call.