support setting acc_steps for batch_sampler

[zhiqiu]

PR Category

Auto Parallel

PR Types

New features

Description

support setting acc_steps for batch_sampler

For example, DP=2, accumulation_steps=4, per_device_batch_size(mini_batch)=1
The original way of auto_parallel will set batchsize=per_device_batch_size*accumulation_steps,

per_device_batch_size = 1
dp_world_size = 2
accumulation_steps = 4
sampler = DistributedBatchSampler(dataset, ..., batch_size=per_device_batch_size*accumulation_steps, ...)

The results of input_ids:
card0: [0,1,2,3], [8,9,10,11], ...
card1: [4,5,6,7], [12,13,14,15], ...

The new way will set batchsize=per_device_batch_size, and sampler._acc_steps=accumulation_steps,

per_device_batch_size = 1
dp_world_size = 2
accumulation_steps = 4
sampler = DistributedBatchSampler(dataset, ..., batch_size=per_device_batch_size, ...)
sampler._acc_steps = accumulation_steps

The results of input_ids:
card0: [0,2,4,6], [8,10,12,14], ...
card1: [1,3,5,7], [9,11,13,15], ...

The two way are right, but the new way can be consistent with dynamic-hand-parallel.
So this PR adds it.
Pcard-76459

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.