prompts are not used in WinRateCallback

[qgallouedec]

          Not from your PR, but I do not think these prompts are ever used. Given the potentially high cost of running this callback, perhaps the user could provide the eval dataset or a subset of it directly, rather than assuming the callback needs to be run on the entire eval dataset.

Example:

...
    def __init__(
        self,
        eval_dataset: Dataset,
        judge: BaseJudge,
        trainer: Trainer,
        generation_config: Optional[GenerationConfig] = None,
        batch_size: int = 4,
    ):
        self.generation_config = generation_config
        self.judge = judge
        self.ref_completions = []
        self.trainer = trainer
        self.eval_dataset = eval_dataset
...

Originally posted by @edbeeching in #1856 (comment)

[qgallouedec]

That's a good point!
I'll address this issue in a dedicated PR

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[qgallouedec]

Still need to be addressed