[DPO] Adding weighted preference optimization (WPO)

[gaetanlop]

What does this PR do?

Adding WPO to the DPOTrainer. The paper is in the list of accepted papers of EMNLP 2024 and introduces an elegant method to similuate on-policy data while training on off policy preference pairs. It works by prioritizing the most probable samples under the current policy during optimization.

Implementation wise, it is independent from the loss function being used as it only provides a weighting for each sample pair during loss computation. My implementation is based on the authors implemation in https://github.com/wzhouad/WPO/blob/main/scripts/run_wpo.py

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Did you write any new necessary tests?

Who can review?

@kashif

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kashif]

@gaetanlop tests for visual-llm models are failing i believe due to an extra argument?

[gaetanlop]

@kashif my bad, should be fixed now

[gaetanlop]

Hey @kashif, I don’t think the failing test is related to the PR. Am I right?

[qgallouedec]

@gaetanlop tests for visual-llm models are failing i believe due to an extra argument?

No, see #2147 (comment)

[qgallouedec]

Please don't merge before #2131, it introduces some modification in the documentation, that I'd like this PR to take into account. I'll review this one in details when #2131 is merged

[gaetanlop]

Looks like #2131 has been merged. I will check if there are any modifications required.

[gaetanlop]

Thanks for the great doc @kashif, after checking #2131, it looks like it doesn't require any updates on this PR