This paper addresses the challenge of data scarcity in Reinforcement Learning (RL) for code editing. To mitigate this, the authors propose Adaptive Ambiguity Replay (A2R), a plug-and-play data module designed to prioritize high-utility training samples. Unlike existing methods that rely on static metrics, A2R leverages training dynamics to identify and replay "ambiguous" samples that are critical for model generalization.
- We implement A2R based on verl. Please follow it to create the environment.
# Configure MODEL_PATH, CKPTS_DIR, TRAIN_FILE, TEST_FILE in run_grpo.py and run_dapo.py before you run.
python -u run_grpo_a2r.py --reward_function edem --model_name Qwen2.5-Coder-7B-Instruct --GPUs 0,1,2,3,4,5,6,7 --rollout_bsz 512 --update_bsz 32 --rollout_n 8 --adaptive_lr_enabled --adaptive_lr_decay_coef 5 --verbose replay-nb-std-alr5 # for grpo
python -u run_dapo_a2r.py --reward_function edem --model_name Qwen2.5-Coder-7B-Instruct --GPUs 0,1,2,3,4,5,6,7 --rollout_bsz 512 --update_bsz 32 --rollout_n 8 --adaptive_lr_enabled --adaptive_lr_decay_coef 5 --verbose replay-nb-std-alr5 # for dapo