support tensor parallel #1599

minhthuc2502 · 2024-01-11T16:25:16Z

WIP for the feature tensor parallel. There are some points to investigate:

Make new version of converter to move forward the number heads before the appearance of weight, bias in self attention to deal with group query attention.
Packaging python wrapper: how to deal with MPI and NCCL when packaging

Update:

Fully support tensor parallel mode. The model could be split between GPUs on the same machine and even on machines different by configuring more network and NFS things.

duydq12 · 2024-01-12T07:37:06Z

LGTM. It helps me a lot. I'm looking forward to seeing the full release version.

minhthuc2502 marked this pull request as draft February 15, 2024 13:39

minhthuc2502 force-pushed the dev/tensor_parallel_nvcc branch 2 times, most recently from 71b440c to 5e9dc28 Compare February 28, 2024 15:21

minhthuc2502 marked this pull request as ready for review March 1, 2024 15:32

minhthuc2502 mentioned this pull request Mar 1, 2024

Splitting LLM layers across multiple GPUs #1052

Closed

minhthuc2502 changed the title ~~tensor parallel by nccl + mpi~~ support tensor parallel Mar 2, 2024

minhthuc2502 added 2 commits March 4, 2024 11:49

tensor parallel support

1da6b31

add docs

ac8f7ae

minhthuc2502 force-pushed the dev/tensor_parallel_nvcc branch from 28473ae to ac8f7ae Compare March 4, 2024 10:49

small fix

05a2702

minhthuc2502 force-pushed the dev/tensor_parallel_nvcc branch from 1663178 to bcefb34 Compare March 4, 2024 16:52

fix adding bias multiple times in layer output.

4ee128f

minhthuc2502 force-pushed the dev/tensor_parallel_nvcc branch from bcefb34 to 4ee128f Compare March 4, 2024 17:04

minhthuc2502 merged commit cec65f1 into OpenNMT:master Mar 5, 2024
17 checks passed

Provide feedback