se_resnet_152 multi-gpu profile

[jacquesqiao]

profile model

https://github.com/PaddlePaddle/models/blob/develop/fluid/image_classification/se_resnext.py

profile with timeline tool

single card



two cards with nccl

two cards without nccl

some simple conclusion:
1, there is a huge gap between forward and backward when nccl is used
2, there is a huge gap between each step.

[jacquesqiao]

#8624
some earlier profile result:

paddle-P40-1-card(无内存优化), max batch size=10, speed(fp+bp): 1.00sec
paddle-P40-1-card(有内存优化), max batch size=25, speed(fp+bp): 1.90sec
paddle-P40-8-card(无内存优化), max batch size=80, speed(fp+bp): 2.35sec
pytorch-P40-1-card, max batch size=48, speed(fp+bp): 2.06sec
pytorch-P40-8-card, max batch size=384, speed(fp+bp): 2.77sec

[tonyyang-svail]

@jacquesqiao could you post the link to the profiling script?

[jacquesqiao]

@tonyyang-svail https://github.com/PaddlePaddle/models/blob/develop/fluid/image_classification/se_resnext.py I use this model.