Complete FP16 (float16) vector support across Lucene

[Pulkitg64]

Description

We have been adding FP16 support at various places in Lucene. Listing them below:

• Add fp16 (float16) vector encoding support, no quantization support: Add fp16 (float16) vector encoding support - no quantization support #16383
• Add scalar quantization support for Fp16 vector encoding: Add scalar quantization support for Fp16 vector encoding #16473
• Fp16 re ranking and prefetch support: Fp16 re ranking and prefetch support #16628 (Still not reviewed and merged)

I think there are few more places where the Fp16 support is still missing from the main branch. This is an overview issue which tracks all the task where we need to add the support. Listing few tasks below which I am able to identify but there might be more:

• DiversifyingChildrenFloat16KnnVectorQuery (lucene/join). Parent-block-join diversifying KNN. Float and Byte siblings exist; no fp16 variant.
• Float16VectorSimilarityQuery (lucene/core). Threshold based vector search. FloatVectorSimilarityQuery and ByteVectorSimilarityQuery exist over AbstractVectorSimilarityQuery but no fp16 subclass.
• Implement scalar quantization support for FP16 vectors in newly added de-duplicating vector format (Also being tracked as part of Improvements to the sandboxed de-duplicating vector format #16595)
• PanamaVectorUtilSupport. short[] dotProduct/cosine/squareDistance currently delegate to the scalar implementation (FLOAT16_DELEGATE). SIMD vectorization is not possible today; it depends on a future JDK release adding Float16Vector support.
• In Faiss codec.

There could be more, so please feel free to add more tasks here.

[rcpeken]

Hi @Pulkitg64, thanks for putting this list together.

I'd like to pick up the Float16VectorSimilarityQuery item, unless you or
someone else already has it in progress. My plan is to mirror
FloatVectorSimilarityQuery (including the Adaptive / Explicit variants)
on top of Float16VectorValues and the short[] search path from #16383, and
add tests alongside the existing float/byte similarity query tests.

As far as I can tell this doesn't depend on #16628, but let me know if you'd
rather have it wait for that PR, or if there's anything specific you'd like the
fp16 variant to handle differently.

[kaivalnp]

Thanks @rcpeken, please feel free to open a PR for Float16VectorSimilarityQuery -- I don't think it depends on the PR you linked.

[rcpeken]

Opened #16735 for the Float16VectorSimilarityQuery item.

It mirrors FloatVectorSimilarityQuery on top of the fp16 APIs from #16383.
One note: I only added the Adaptive variant, since Explicit is deprecated in the float/byte queries and only exists for 10.4 compatibility. Happy to add it if you'd prefer full parity.

@kaivalnp would appreciate a review when you have time.