ParallelDo performance on VGG

[tonyyang-svail]

Major take aways:

1. Parameter copy is still a big bottleneck (large net large VGG16, Memcpy takes up to 80%)
2. We do need multiple streams (AllReduce Kernel takes up 70% of the total kernel time)
3. NCCLInit should not be called at every iteration. It takes about 70ms for one GPU and 90ms for four GPUs.

Background

test script, command line

Net: VGG16
Model Size: 409M (The original definition of VGG16 net is incorrect #8718)
Batchsize: 16 for each GPU
BatchNorm: OFF. Since parallel_do doesn't support this.

Inputs are randomly generated on each GPU. So no overhead on copying training data to different devices

Result

Time unit: milliseconds.

# GPUs	copy weights (ms)	forward and backward (ms)	merge gradient (ms)	apply gradient (ms)	total
1	N/A	130	N/A	5
1 NCCL in backward	N/A	220	N/A	5
4	350	130	350	5
4 NCCL in backward	350	650(AllReduce takes about 70%)	N/A	5

[panyx0718]

I observe the same issue: ncclAllReduce is obviously taking too much time. ParameterCopy and ncclInit are similar issue.
Another thing worth noticing is: I see gap between different training step, which could be due to python inputs.

I'm surprised and concerned that ParallelDo doesn't support BatchNorm. It seems the resnet multi-gpu version has batchnorm? @jacquesqiao

[jacquesqiao]

@panyx0718 ParallelDo support batch_norm, this problem is fixed by #8652