Will this book talk about RLHF?

[jingedawang]

Great book! I read all the notebooks in this repo and here is a question.

I heard RLHF(Reinforcement Learning from Human Feedback) is the core technique of ChatGPT. Is this book talking about it?

I see there will be an extra material about dpo on preference fine-tuning. Is it equivalent to RLHF? What's the popular practice in the current industry after instruct-finetuning?

Thanks!

[rasbt]

Thanks! Regarding DPO, I've actually implemented it for Chapter 7, but then I removed it for 2 reasons

1. The chapter got way, way too long and exceeded the page limits
2. I am not very happy with the DPO results

DPO is a nice, and relatively simple technique for preference finetuning, but it didn't quite satisfy the bar regarding fundamental and established techniques that work well. I am currently going to be busy with finishing up the work on the book itself in the next few weeks, but then I plan to polish up the DPO part and perhaps either share it here or on my blog. Then, I plan to do the same with RLHF with dedicated reward models.

In the meantime, you might like my two articles here:

• How Good Are the Latest Open LLMs? And Is DPO Better Than PPO?

• Tips for LLM Pretraining and Evaluating Reward Models

[jingedawang]

Thank you for answering! One more question, towards your plan about DPO and RLHF, are they the same thing? Or the RLHF only refers to PPO?

[rasbt]

Some people describe DPO as a technique for RLHF, but since the rewards are implicit, I would say it's maybe its own thing.

[jingedawang]

Got it. Looking forward to your next work about DPO and RLHF!

[d-kleine]

Yeah, I think it would be nice to include something related to RLHF in the bonus materials.