Deprecate PPOTrainer

[qgallouedec]

# v0.11 <- wer're here
from trl import PPOTrainer  # Old implementation, raise DeprecationWarning("PPOTrainer is deprecated and will be removed in a future release. Please use PPOv2Trainer instead.")
from trl import PPOv2Trainer # New implementation

# v0.12
from trl import PPOTrainer # New implementation
from trl import PPOv2Trainer # New implementation, raise DeprecationWarning("PPOv2Trainer is deprecated and has been renamed to PPOTrainer. Please use PPOTrainer instead.")

# v0.13
from trl import PPOTrainer # New implementation
from trl import PPOv2Trainer # ImportError("PPOv2Trainer has been renamed to PPOTrainer. Please use PPOTrainer instead.")

• Deprecate PPOTrainer
• Deprecate PPOConfig
• Update the doc (replace all examples containing PPOTrainer) -> will do in another PR

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Sino-Huang]

I would like to ask why original PPO script used AutoModelForCausalLMWithValueHead class to obtain policy model but now PPOv2 script uses AutoModelForCausalLM class. Are they interchangeable? I observed that if I continue using AutoModelForCausalLMWithValueHead class in PPOv2, I am able to have better training speed if I have LoRA config.

Also LoRA + PPOv2 example seems missing.