DeepSpeed MoE #1310
DeepSpeed MoE #1310
Conversation
Co-authored-by: Alex Muzio <Alex.Muzio@microsoft.com> Co-authored-by: Alex Muzio <alferre@microsoft.com> Co-authored-by: Ammar Ahmad Awan <ammar.awan@microsoft.com> Co-authored-by: Conglong Li <conglong.li@gmail.com> Co-authored-by: Felipe Cruz Salinas <Andres.Cruz@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Reza Yazdani <44502768+RezaYazdaniAminabadi@users.noreply.github.com> Co-authored-by: Reza Yazdani <reyazda@microsoft.com> Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: Shaden Smith <shaden.smith@microsoft.com> Co-authored-by: Young Jin Kim <youki@microsoft.com> Co-authored-by: alexandremuzio <ax.muzio@gmail.com> Co-authored-by: bapatra <bapatra@microsoft.com>
awan-10
requested review from
cli99,
conglongli,
eltonzheng,
jeffra,
minjiaz,
niumanar,
RezaYazdaniAminabadi,
samyam,
ShadenSmith and
tjruwase
as code owners
| @@ -23,7 +23,8 @@ void Adam_Optimizer::Step(float* _params, | |||
| float* _exp_avg, | |||
| float* _exp_avg_sq, | |||
| size_t _param_size, | |||
| __half* dev_params) | |||
| __half* dev_params, | |||
| bool half_precision) | |||
There was a problem hiding this comment.
@awan-10, @jeffra, @RezaYazdaniAminabadi, sorry i realize this is almost 3 years old, but I need to understand the introduction of half_precision in this PR. Who can I talk to you? Thanks!
For context, this affects a current PR under review
#5409
There was a problem hiding this comment.
Hi @tjruwase
I can chat with you on this.
I think this is mostly added here to make sure the right AVX operation is selected for FP32 vs FP16. However, as i see it is now templated in this new PR.
Thanks,
Reza
There was a problem hiding this comment.
@RezaYazdaniAminabadi, thanks for the response. Your explanation was my guess as well. I think the entire code can be greatly simplified with the template usage and also improved type support in torch. Can you please help to review the new PR and also engage in the conversation there?
This PR introduces DeepSpeed Mixture of Experts (MoE) support. The code has been written in collaboration with many contributors at Microsoft including the Z-code team.