CPU memory usage for PixelReasoner

[zulihit]

Hi, thank you for open-sourcing verl-tool and the PixelReasoner example!

I'm trying to reproduce the PixelReasoner RL training with the 3B Qwen2.5-VL SFT model:

• Script: examples/pixel_reasoner GRPO config, slightly adapted

However, I am hitting very high CPU memory usage and Ray keeps killing workers due to memory pressure (OOM), even though GPU memory usage is quite low.

Given that PixelReasoner involves images, videos, and long reasoning chains, I suspect the current default recipe is close to the upper bound of what my machine can handle. I’d like to ask:

For this kind of multi-modal long-context RL setup, how should I configure the hyperparameters (batch size, concurrency, lengths, async reward, etc.) to better fit my hardware?

My server specs:

• GPUs: 8 × A100, each with ~48 GB VRAM
• CPU RAM: 503 GiB
• Shared memory (/dev/shm): 128 GiB
• Running inside a Docker container, shm mounted as 128G

I’m running (simplified bash script):

rl_alg=grpo
n_gpus_per_node=8
n_nodes=1
n=8
batch_size=128
ppo_mini_batch_size=64

max_prompt_length=16384   
max_response_length=16384
max_obs_length=8192   
gpu_memory_utilization=0.5
do_offload=True

PYTHONUNBUFFERED=1 python3 -m verl_tool.trainer.main_ppo \
    data.dataloader_num_workers=0 \
    actor_rollout_ref.rollout.max_num_seqs=16 \
    actor_rollout_ref.agent.max_concurrent_trajectories=16 \
    trainer.val_before_train=False \
    

[cjakfskvnad]

Have you tried do offload=false?