Pixel Reasoner Tool Call Frequency Gradually Approaches Zero in Verl-tool

[LXDxmu]

Thanks for opening such wonderful work!

I'm currently reproducing the Pixel Reasoner using the official verl-tool implementation https://github.com/TIGER-AI-Lab/verl-tool with a single-node setup, following the default training script at: https://github.com/TIGER-AI-Lab/verl-tool/blob/main/examples/train/pixel_reasoner/train_qwen25vl.sh

During training, I observed two key issues:

1. Tool Call Frequency Decline: The pixel reasoner tool's call frequency progressively decreases over time and eventually approaches zero.

2. Both the curiosity reward and penalty reward remain consistently at 0 throughout the entire training process.

Could you please advise if this is expected behavior in verl-tool? (Note: We found that without tool usage, the performance can still reach around 84% accuracy.) If not, what might be causing this issue and how can I resolve it within the verl-tool implementation?

Additionally, would it be possible to share reference training curves showing the Pixel Reasoner tool usage pattern in verl-tool for comparison? Here's my current training curve for reference.

[cjakfskvnad]

I guess something in your setting is wrong. If tool rate is not zero at the beginning, the curiosity reward should not remain zero throughout the training: at least not in the beginning.

[Lyman-Smoker]

Hi @LXDxmu ,

I am facing the similar issue when trying to reproduce Pixel-Reasoner with verl-tool implementation.

Have you solved this issue? Could you please share your solutions?

Thanks for your help in advance :)