MOD-19169: Use native FP16 arithmetic for SQ8–FP16 inner product

[dor-forer]

SQ8–FP16 inner product currently widens SQ8 codes and FP16 query values to FP32 on every distance call. This adds native AVX-512 FP16 multiplication and accumulation for inner product and normalized cosine.

For dimensions 128–255 the tier uses one FP16 accumulator. From dimension 256 it uses four independent accumulators, merges them pairwise in FP16, then widens the merged lanes for FP32 horizontal reduction. Dimension and ISA selection happen when choosing the distance function. The query and storage representation stay unchanged; scale, offset and query-sum metadata remain FP32. Both nonfinite-result and non-nearest-MXCSR guards retain the existing mixed-kernel fallback.

The tier requires AVX512F/FP16/BW/VL and respects the existing upper dimension limit. Smaller dimensions, unsupported x86 CPUs, Arm, L2 and regular dense FP16 retain their existing paths. No benchmark workflow changes are included.

Precision changes because the code-value dot product is accumulated in FP16 before FP32 affine correction. Four accumulators shorten half accumulation chains, and FP32 reduction avoids half rounding during horizontal addition. This does not make the result equivalent to fully decoded FP32 distances or guarantee unchanged recall for every query.

Validation and measured tradeoff

Three-way Intel validation, Xeon Platinum 8488C, identical saved SQ8 graphs, 1,000 queries per dataset, k=100, ef=200:

Dataset	Original one-sum PR: ms / recall	Four sums, FP16 reduction	Four sums, FP32 reduction (selected)
Fashion, 512 dimensions	1.1182 / 98.090%	1.0900 / 98.104%	1.1164 / 98.833%
DBpedia, 768 dimensions	1.1733 / 99.395%	1.1430 / 99.423%	1.1605 / 99.465%

The selected variant improves aggregate recall by 0.743 and 0.070 percentage points over the original one-sum PR. Latency differences (−0.17% and −1.09%) are inside baseline process spreads (6.40% and 14.76%): this run does not establish an end-to-end speedup over that version. Recall improves at all measured ef values (100/200/400); individual queries can still lose hits. At ef200, 74 Fashion and 81 DBpedia queries lose hits, while 582 and 150 gain. DBpedia has one brute-force cutoff tie; Fashion has none.

Dependency-bound direct-kernel measurements at dimensions512/768 improve from 49.009/60.972 ns to 35.280/40.887 ns (−28.01%/−32.94%). This kernel gain is not an HNSW latency claim. Several hot-throughput cases and the 240 MiB pool have substantial variability. The dimension128 cutoff is conservative, not a proven universal crossover: hot throughput and dependent-call timings favor different implementations near the boundary.

Evidence and scope:

• Final production-source Intel/Arm validation built exact commit 742193c4daef41666c9c74e30e06fcea0f420aac, without experiment macros. Intel Xeon Platinum 8488C: 1,601 spaces + 52 components + 209 SQ8 HNSW tests passed, including all seven native cases; zero skipped/disabled. Arm Neoverse-V2: 1,556 + 52 + 209 passed, zero skipped/disabled (x86 native suite is compiled out). Both tier-linkage checks and both EC2 shutdown jobs passed. Source commit/tree/clean checkout, CMake caches, CPU features, test-binary SHA256 records and raw XML were checked. This closes the native execution gap between the experimental benchmark source and the production source.
• Seven interleaved processes (baseline/baseline/four/reduce/reduce/four/baseline), three repetitions at each ef; reported latency is the median of repeated mean query latencies, including preprocessing. Reference is exhaustive FP32 brute force over original FP16 values. All process input/graph/reference identities match; reference-cache hashes were recomputed.
• Both candidates passed 1,601 spaces, 52 components and 209 SQ8 HNSW tests, including all seven native tests, plus tier-linkage checks. Tests independently model half rounding, all residual/leftover paths, dispatch boundaries, FP32 metadata, overflow and directed-rounding fallback. No tolerance relaxation.
• The kernel sweep contains 1,890 raw timings across 22 dimensions, hot/dependent cases and a full shuffled 240 MiB pool. Distinct binary hashes and expected dispatch paths were verified.
• Saved-graph results cover query execution. Graph construction with this final arithmetic, Arm performance and an uncompressed-FP16 comparison were not measured. Earlier one-sum construction and mixed-versus-native timings do not validate these final variants directly.
• Earlier production PR CI passed for head 25968215c721c25847588e14e3d1503a0b794d38 using its PR merge checkout: formatting, 3,102 normal CTest passes plus 108 skips, 3,134 sanitizer passes plus 108 skips, and 240 flow passes plus 10 skips. Six ISA-dependent native cases were skipped on the hosted runners; all seven execute in the exact-source Intel validation above. Coverage and CodeQL remain skipped while this PR is a draft.

For MOD-19169. Query quantization in #1072 remains deferred. The PR remains draft and is not merged.

Follow-up cleanup centralizes thresholds 128/256, names the paired four-sum/FP32-reduction flag, and renames the load helper to LoadQuantizedValues. The kernel now delegates accumulation, horizontal reduction, and metadata correction to three inline helpers; the rounding and nonfinite fallback checks remain in the entry point. The byte pointer is named quantized_values, and the leftover full blocks use explicit remaining >= 32/64/96 checks. Arithmetic and operation order are preserved. The exact final commit passed Intel/Arm validation above; performance benchmarks were not rerun for this refactor.

---

Note

Medium Risk
Changes numeric behavior on the hot SQ8–FP16 distance path for large dimensions on x86, though overflow and rounding-mode fallbacks limit exposure; extensive new tests cover dispatch and arithmetic edge cases.

Overview
Adds a new AVX-512 FP16/BW/VL SIMD tier for SQ8 storage × FP16 query inner product and cosine (dimensions 128–65504 on CPUs with AVX512F, FP16, BW, and VL). The dot product is accumulated in native FP16 (32-wide chunks); from dimension 256 it uses four parallel accumulators with FP32 horizontal reduction, while FP32 scale/offset/query-sum correction is unchanged.

Build/dispatch: OPT_AVX512_FP16_BW_VL gates a new AVX512FP16_BW_VL object file; IP_SQ8_FP16_GetDistFunc / Cosine_SQ8_FP16_GetDistFunc prefer this tier over the existing AVX512F mixed-FP32 path when eligible. Safety: non-nearest MXCSR or non-finite FP16 dot results fall back to Choose_SQ8_FP16_IP_implementation_AVX512F.

Tests add FP16-rounding reference models, dispatch boundary checks, overflow/cancellation cases, and integration with the existing ISA tier walk.

^{Reviewed by Cursor Bugbot for commit 742193c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.55%. Comparing base (1e9b74c) to head (742193c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1074      +/-   ##
==========================================
+ Coverage   97.48%   97.55%   +0.07%     
==========================================
  Files         142      144       +2     
  Lines        9251     9420     +169     
==========================================
+ Hits         9018     9190     +172     
+ Misses        233      230       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.