wrong objective/entropy in RLOOTrainer

[serendipity800]

System Info

newest trl version

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

mean_entropy = (-logprobs).sum(1).mean()

This line is wrong, the -logprobs is obtained by:
logprobs = torch.masked_fill(logprobs, padding_mask, INVALID_LOGPROB)
ref_logprobs = torch.masked_fill(ref_logprobs, padding_mask, INVALID_LOGPROB)
where INVALID_LOGPROB = 1.0
This will cause objective/entropy < 0 on long padded seqs

Expected behavior

The entropy calculation should be right

[asparius]

This issue appears in PPO as well. This was introduced in 0.12.1 which is the rewritten version of the PPO. In the old version, masked_mean is used. I have also checked the PR and change log, there was no mention of this. @qgallouedec can enlighten us.

[qgallouedec]

Should be now fixed with #3801