MOD-14960: Add HNSW SQ8 serialization and microbenchmarks

[dor-forer]

Describe the changes in the pull request

Ports ARM-software/VectorSimilarity-for-Arm#6: V5 SQ8 HNSW snapshots and FP32/FP16 single/multi tiered benchmarks for memory, add/delete, TopK, and range. Unquantized snapshots retain V4 writes and legacy V3 reads.

The port also preserves the trained mean when saving, updates the converter for current Python bindings, and normalizes cosine inputs for IP SQ8 snapshots. SQ8 registrations reuse the existing TopK and range helpers; range recall measures label overlap. Full DBpedia/fashion snapshots for FP32/FP16 are included in the existing download manifests. These benchmarks use prebuilt backends and do not measure training.

Initial validation: Intel targeted suites passed (2,048 tests, one unsupported-hardware skip), all four basics binaries built, four converter integration cases (including absolute-radius checks) passed, and all eight SQ8 add/delete smoke cases passed. All four Intel query/memory configurations completed three repetitions. Arm targeted suites passed (2,011 tests, zero skips), all eight requested targets built, and the single-FP32 query/memory run completed three repetitions using SHA256-identical inputs. Both Arm single-FP32 SQ8 add/delete smoke cases passed (1,024 iterations each).

After the review cleanups, all four FP32/FP16 single/multi benchmark targets rebuilt on Intel and Arm, and all 153 SQ8 tests passed on each architecture. All four full snapshots were regenerated using the current converter, passed save/reload integrity and vector-count checks, and passed 20 Intel smoke cases covering memory, TopK, range, add and delete. Anonymous downloads from the benchmark bucket match the generated SHA-256 checksums.

The full-data timing results below use DBpedia FP32 files that are byte-for-byte identical to the newly generated snapshots. The new smoke checks are functional validation, not replacement timing measurements.

Sample benchmark results

DBpedia single FP32: 1M vectors, dimension 768, M=64, efConstruction=512. Three repetitions, mean range times; radius=0.35, epsilon=0.01, ten queries/repetition.

Host	Unquantized range ms/query	SQ8 range ms/query	Unquantized recall	SQ8 recall
Intel i4i.4xlarge	67.148	50.003	99.9552%	99.7775%
Arm r8g.2xlarge	56.029	44.599	99.9552%	99.7775%

Tiered index allocator memory was 3.493 GiB unquantized and 1.361 GiB SQ8 on both hosts. Range compares unquantized HNSW with SQ8 tiered HNSW over prebuilt backends, at matching settings rather than equal recall. Shared queries are normalized outside timing. TopK uses eight workers and 10 ms completion polling; no small TopK speedup is claimed. These measure index choices, not before/after-port kernel performance or training.

Intel uses GCC 13.3 and the prebuilt SVS dependency; Arm uses GCC 12.3 and the pinned SVS source dependency. The host comparison includes these configuration differences.

Which issues this PR fixes

1. Microbenchmark portion of MOD-14960: Benchmarks for HNSW SQ8 (micro + E2E). The ticket remains open for the accumulation-to-backend transition workload, vector-db-benchmark E2E matrix, and repeatable regression thresholds.

Main objects this PR modified

1. HNSW serializer/factory and quantization mean metadata.
2. Tiered FP32/FP16 benchmark fixtures and registrations.
3. SQ8 conversion script, setup documentation, and serialization tests.

Mark if applicable

• This PR introduces API changes
• This PR introduces serialization changes

---

Note

High Risk
Introduces a new on-disk format and load/save paths for quantized HNSW; incorrect mean/metric/normalization handling would break index fidelity across restarts.

Overview
Adds HNSW encoding V5 so SQ8 indexes can be saved and reloaded with quantType and an optional mean vector, while unquantized indexes still write V4. Saving no longer throws for quantized indices; write version is chosen via getWriteVersion().

Factory/load path reads V5 metadata, builds SQ8 components (FP32/FP16, L2/IP), rejects unknown quantizers, and documents is_normalized=true for pre-normalized cosine SQ8 snapshots. Test builds keep quantization metadata on the index for round-trip checks (replacing the old “reject SQ8 serialization” test).

Microbenchmarks gain standalone and tiered SQ8 HNSW fixtures (DBpedia/fashion), V5 download lists, query normalization for IP-style cosine SQ8, tiered TopK/range/memory/add/delete registrations, set-based range recall, and a convert_to_sq8.py helper plus benchmark docs for regenerating *-sq8.hnsw_v5 files.

^{Reviewed by Cursor Bugbot for commit 226ddda. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ dor-forer
❌ xiangze-arm
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 98.48485% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.56%. Comparing base (f06d036) to head (226ddda).

Files with missing lines	Patch %	Lines
src/VecSim/index_factories/hnsw_factory.cpp	98.07%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1062      +/-   ##
==========================================
- Coverage   97.58%   97.56%   -0.02%     
==========================================
  Files         142      142              
  Lines        9095     9148      +53     
==========================================
+ Hits         8875     8925      +50     
- Misses        220      223       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dor-forer]

Superseded by the smaller stacked PRs: #1064 for SQ8 serialization/load support and #1065 for the microbenchmarks, converter, manifests, and documentation.