Improve judges

[qgallouedec]

Motivation:

Judges aren't always pairwise
Judges may be slow when called with numerous prompts/completions
BaseAPIJudge doesn't provide any simplification

Solutions

1. Define a new inheritance structure:

• BaseJudge for all judges
• BaseRankJudge(BaseJudge) for judges that returns ranking
• BasePairwiseJudge(BaseJudge) for judges that compare two completions and returns the best index

2. We rename HuggingFaceJudge to HfPairwiseJudge and OpenAIJudge to OpenAIPairwiseJudge

3. We use multithreading for api call to speedup inference. Why multithreading instead of asyncio?

	Sequential	Multi-threading	asyncio
InferenceClient	Slow	Ok	Slow (AsyncInferenceClient)
OpenAI	Slow	Ok	Ok (AsyncOpenAI)

4. Other minor changes in the documentation

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[edbeeching]

Another comment unrelated to this exact PR, but perhaps we can include this change. In the original version of the WinRateCallback, it was possible to use the starting model as the reference directly, without having the following:

    if not hasattr(trainer, "ref_model"):
        raise AttributeError("Trainer must have a `ref_model` attribute.")

and the model rather than ref_model was used here:

    def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
        tokenizer = kwargs["tokenizer"]
        tokenizer.padding_side = "left"
        accelerator = self.trainer.accelerator
        with accelerator.split_between_processes(self.eval_dataset["prompt"], apply_padding=True) as prompts:
            self.ref_completions = self.generate_completions_for_model(self.trainer.ref_model, tokenizer, prompts)

This enabled us to use this callback to monitor progress during an SFT, without the need for a ref model.

I believe the result should be the same even in a DPO setting. @qgallouedec are you aware of why this was changed in @lewtun's PR?

[edbeeching]

Hey @qgallouedec, regarding asyncio vs multithreading. The original version of judges used threadpools, do you know why this was changed to asyncio in the first place? Perhaps there is a good reason we are not aware of.

[qgallouedec]

why this was changed to asyncio in the first place?

asyncio is a more natural choice when handling I/O-bound tasks because it efficiently manages a large number of concurrent operations with lower overhead, avoids the complexities of thread synchronization, and offers better scalability and performance for tasks like API requests.

Also, I wanted to remove BaseAPIJudge, so this was the opportunity to question threadpool's choice.

[edbeeching]

Thanks for creating issues for the points I raised. LGTM barring the remaining minor comments and resolving the remaining conflicts.