grpo inference error

[jiangyuan1018]

The model qwen7b-instruct crashes suddenly within a fixed range of steps during grpo training, and then outputs only exclamation marks!!!
trl =0.16.1
vllm =0.72

[luolun]

请问解决了吗，我用trl训练也遇到只输出感叹号的问题

[jiangyuan1018]

猜测是vllm版本造成问题

[lewtun]

This looks like a form of reward hacking to me: do you see a big spike in the completion lengths as well?

[jiangyuan1018]

I had indeed suspected this to be a case of reward hacking initially. However, even when I excluded length from the reward considerations, the phenomenon persisted. The issue only manifests when using vllm for inference - it completely disappears with alternative inference methods.