Reranker dynamic quantization

[rawsh-rubrik]

Feature request

--dtype support for rerankers

Motivation

Easily quantize cross encoder models

Your contribution

. (will look into this)

[michaelfeil]

Correct, that is currently not possible, but easy to implement. You are welcome to contribute this.

Task:

• Add a similar integration as in: https://github.com/michaelfeil/infinity/blob/65afe2b3d68fda10429bf7f215fe645be20788e4/libs/infinity_emb/infinity_emb/transformer/embedder/sentence_transformer.py#L87C9-L90C14
• add a test (verifying that the quantization of a reranker model works). Please use BGE-reranker BASE or mixedbread-xsmall reranker for the test