special deal with the inplace case in auto parallel pass.

[winter-wang]

PR Category

Auto Parallel

PR Types

New features

Description

• support inplace case in auto parallel pir pass.
  • as follows, the partition pass will insert pd_op.assign_out_ op for inplace value.

  .........
  (%930) = "dist_op.reshard" (%3) {.......}
  (%931) = "dist_op.reshard" (%2) {.......}
  
  # 其中 %930和%933， %931和%934为相应的inplace变量，即在算子执行中会被修改。
  (%932, %933, %934, %935, %936, %937) = "pd_op.adamw_"  (%108, %655, %656, %930, %931,,,,,,,,,,,) 
  
  # %930在pd_op.adamw_中发生了变化，因此需要同步到原始的%3变量
  (%938) = "dist_op.reshard"(%930) {.......}
  "pd_op.assign_out_"  (%938, %3) {.......}
  # %931在pd_op.adamw_中发生了变化，因此需要同步到原始的%2变量
  (%939) = "dist_op.reshard" (%931) {.......}
  "pd_op.assign_out_"  (%939, %2) {.......}

• add distribute branch for assign_out_ op.
• replace "pd_op.c_allreduce_sum_" with "pd_op.c_allreduce_sum" in p_to_r_reshard_func. Because the semantics of reshard op is non-inplace。
• alignment the precision for llama demo with 8 card 3D mix parallel.

Other

Pcard-67164

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[JZ-LIANG]

LGTM