[feat] Add query prompts to Information Retrieval Evaluator

[ArthurCamara]

Currently, there is no way to add prompts to the Evaluators. This PR adds this function to the Information Retrieval Evaluator, as discussed in #2848 (comment)

---

Older description that need to go another PR:

Most models, specially based on Mistral, are trained with left-padding in the tokenizer, instead of the common right-padding when evaluating. (e.g., LLM2Vec), but this is not covered in the current prompt_length strategy.
Currently, If a batch with mismatched lengths is left-padded, the masking will mostly mask out padding tokens. This PR fixes that by adding a mask_prompt argument to the tokenize function. When this flag is set, the code tries to find the first non-padding token in each sentence, and will mask everything between that and prompt_length, adding a prompt_mask representation to the output dictionary.

Perhaps a more "elegant" solution would be to replace prompt_length entirely, but this could break Instructor models.

[ShengYun-Peng]

Hi @ArthurCamara, I'm reading the description of the PR and curious whether we need left padding in the text embedding models. I was also thinking about adding a left_pad argument following the LLM training procedure. But we may not need left_pad because text embeddings are not language modeling, and it only requires one forward pass to fetch the embedding, while language modeling typically requires multiple forward passes. Thus, there shouldn't be any gap (pad tokens) between the prompt and generated output in language modeling, leading to the left_pad operation. However, text embedding can take padding either on left or right, thus I think right_pad should already satisfy current text embedding models. Lmk your thoughts on this and correct me if I am wrong.

[ArthurCamara]

Hi @ArthurCamara, I'm reading the description of the PR and curious whether we need left padding in the text embedding models. I was also thinking about adding a left_pad argument following the LLM training procedure. But we may not need left_pad because text embeddings are not language modeling, and it only requires one forward pass to fetch the embedding, while language modeling typically requires multiple forward passes. Thus, there shouldn't be any gap (pad tokens) between the prompt and generated output in language modeling, leading to the left_pad operation. However, text embedding can take padding either on left or right, thus I think right_pad should already satisfy current text embedding models. Lmk your thoughts on this and correct me if I am wrong.

Hi @ShengYun-Peng! I agree that, in theory, either left or right-padding should be fine when training a text embedding model. However, there are some cases, specially when training a decoder-only model that uses causal attention (like most LLMs today), we usually want to use the <EOS> token as the sentence representation, instead of the first (e.g., the [CLS] token in BERT`) or the average of all tokens. This is really hard (and inefficient) to do if the batch sentences are right-padded, as the last token is now in some random position.

This is the approach taken by, for instance, E5-Mistral and the NV-Retriever models.
If you are not using the last token as the sentence representation (say, you have modified your attention mechanism to be bi-directional, like the NV-Embed-v2 model), then you should be fine using right padding.

[ArthurCamara]

Hi @ArthurCamara, I'm reading the description of the PR and curious whether we need left padding in the text embedding models. I was also thinking about adding a left_pad argument following the LLM training procedure. But we may not need left_pad because text embeddings are not language modeling, and it only requires one forward pass to fetch the embedding, while language modeling typically requires multiple forward passes. Thus, there shouldn't be any gap (pad tokens) between the prompt and generated output in language modeling, leading to the left_pad operation. However, text embedding can take padding either on left or right, thus I think right_pad should already satisfy current text embedding models. Lmk your thoughts on this and correct me if I am wrong.

Hi @ShengYun-Peng! I agree that, in theory, either left or right-padding should be fine when training a text embedding model. However, there are some cases, specially when training a decoder-only model that uses causal attention (like most LLMs today), we usually want to use the <EOS> token as the sentence representation, instead of the first (e.g., the [CLS] token in BERT`) or the average of all tokens. This is really hard (and inefficient) to do if the batch sentences are right-padded, as the last token is now in some random position.

This is the approach taken by, for instance, E5-Mistral and the NV-Retriever models. If you are not using the last token as the sentence representation (say, you have modified your attention mechanism to be bi-directional, like the NV-Embed-v2 model), then you should be fine using right padding.

That being said, I've just noticed an issue in the code that will be harder to fix than I expected. So, I'm reverting these changes and keeping only the prompts to the evaluators in this PR for now.

[tomaarsen]

That being said, I've just noticed an issue in the code that will be harder to fix than I expected. So, I'm reverting these changes and keeping only the prompts to the evaluators in this PR for now.

Sounds good!
I'm glad that most "LLM-converted embedding models" use last-token pooling, as it's comparatively easy to implement.

I think this PR looks good at a glance. Do you want to mark it ready for review?

• Tom Aarsen

[tomaarsen]

Thanks a bunch for this! I appreciate it.

• Tom Aarsen

[ShengYun-Peng]

Hi @ArthurCamara, I'm reading the description of the PR and curious whether we need left padding in the text embedding models. I was also thinking about adding a left_pad argument following the LLM training procedure. But we may not need left_pad because text embeddings are not language modeling, and it only requires one forward pass to fetch the embedding, while language modeling typically requires multiple forward passes. Thus, there shouldn't be any gap (pad tokens) between the prompt and generated output in language modeling, leading to the left_pad operation. However, text embedding can take padding either on left or right, thus I think right_pad should already satisfy current text embedding models. Lmk your thoughts on this and correct me if I am wrong.

Hi @ShengYun-Peng! I agree that, in theory, either left or right-padding should be fine when training a text embedding model. However, there are some cases, specially when training a decoder-only model that uses causal attention (like most LLMs today), we usually want to use the <EOS> token as the sentence representation, instead of the first (e.g., the [CLS] token in BERT`) or the average of all tokens. This is really hard (and inefficient) to do if the batch sentences are right-padded, as the last token is now in some random position.

This is the approach taken by, for instance, E5-Mistral and the NV-Retriever models. If you are not using the last token as the sentence representation (say, you have modified your attention mechanism to be bi-directional, like the NV-Embed-v2 model), then you should be fine using right padding.

Thank you so much for the explanation! I agree that if we are using the eos token for embedding, left padding will make it easier.