LAMB is not suited for tensor parallel #22
Description
The current LAMB optimizer implementation does not support tensor parallel as it needs to compute norm of the whole matrix. It is not compatible with tensor parallel as the tensor is split.
{{ message }}