Implementing multi-GPUs Training for RecBole #961
Conversation
|
@juyongjiang Hi, thanks for your PR and we will carefully check it. |
| batch_size=dataset.shape[0], | ||
| sampler=DistributedSampler(dataset, shuffle=False)) | ||
| for data in dis_loader: | ||
| batch_data = data |
There was a problem hiding this comment.
I do not understand this loop, it seems batch_data will be the last 'data' of dis_loader, could you please explain it?
There was a problem hiding this comment.
And did you test your code in some datasets like ml-100k? Could you provide us the performance results of models? I want to know if the model performance will change a lot compared with single-GPU training.
There was a problem hiding this comment.
Hi, Xingyu! Yeah, my pleasure! In our DataLoader class, I assign batch_size=dataset.shape[0] that means it extracts all data in current batch_size. So the length of dis_loader will be only one, i.e. like this for data in range(1).
There was a problem hiding this comment.
Yeah, of course! Please wait for a moment! I will provide a table to illustrate the performance compared with single GPU training.
There was a problem hiding this comment.
@2017pxy Hi, Xingyu! I have got the experimental results. It seems that it doesn't decrease the performance a lot but significantly reduces the training time by about 3.78 times. BTW, I just run the experiment only one time. So I think this performance drift can be ignored. : )
Note that the original item means I got the result through running your original RecBole code. And multi-GPUs item result is produced by 3 multi-GPUs.
There was a problem hiding this comment.
@2017pxy Any further questions and or comments? Thanks in advance.
There was a problem hiding this comment.
Hi @juyongjiang @hunkim, sorry for late reply.
Following your implementation, our team modified the trainer and made some tests. We find your implementation works well for model training. Thanks for your contribution!
However, since the time cost of run_recbole is mainly from model evaluation, we want to implement the multi-GPUs evaluation as well and release together with the multi-GPUs training. Unfortunately, we face some problems when we apply your implementations to evaluation since the data organization for evaluation is different. Thus, I am sorry to tell you that it still takes some time to release this new feature, and even this new feature might not be added in next version.
Thanks again for your implementation, and if you have any idea or suggestions about multi-GPUs evaluation, please let us know.
There was a problem hiding this comment.
@2017pxy Okay, got it! Thanks for your reply. I will implement the multi-GPUs evaluation as well and pull a new request. : )
|
@juyongjiang Cool! |
|
Hello， I used your methods to implement multi-gpus on kagt, but after setting |
Hi there,
I am a fan of the RecBole framework. Considering the complexity of the RecBole framework, I provide an easy but feasible method to achieve multi-GPUs training.
The core implementation idea is that re-wrapping the internal data type of
Interactionto the PyTorchDataloaderobject. The more details in my pull request branch "fix_multi_gpus", please check it.Note that it is just one of the promising ways to realize multi-GPUs training. Hoping this method can inspire you to come up with a more novelty method to do it.
To use multi-gpus training model (e.g. BERT4Rec), you just need to run the following command:
multi_gpus: Truein yourconfig.yamlfiles.$ python -m torch.distributed.launch --nproc_per_node=3 run_recbole.py --model=BERT4Rec --config_files recbole/properties/model/BERT4Rec.yamlBest Regards,
John