Distributed training VGG16 benchmark questions

[helinwang]

In Distributed training VGG16 benchmark

• Does Single Node Single Thread mean single node, if so, the PServer Count: 10, Trainer Count: 20 below better be removed, since it's confusing.
  • Answer: yes it means single node, removed the confusing part from the doc.
• In Different Batch Size, there is Per trainer CPU Core: 1, why only 1 core is used, we probably should be utilizing all the computing resource of one node.
  • Answer: actually it means the test is using MKL_NUM_THREADS=1 (doc is now updated), the entire CPU is available for the trainer process. Detail please see here. @typhoonzero will update the GPU results.
• Does 78.64% in Accelerate Rate means the performance compared to ideal scenario?
  • Answer: yes.
• In Accelerate Rate, PaddlePaddle v2 is much better than Fluid in the trainer count = 20 case, do you know why is it so? Same question for the metric in Different Pserver Count, V2 is much better than Fluid.
  • Answer: No, need to figure it out.

[typhoonzero]

In Different Batch Size, there is Per trainer CPU Core: 1, why only 1 core is used, we probably should be utilizing all the computing resource of one node.

Currently multicore uses MKL threading which is much like OMP, I'd prefer to test with parallel_do support. By the way, if we want to see whether the network communication performance will become a bottleneck, GPU tests may be easier to hit the point.

Does 78.64% in Accelerate Rate means the performance compared to ideal scenario?

Yes. Which means actual performance / num_nodes * single thread result.

In Accelerate Rate, PaddlePaddle v2 is much better than Fluid in the trainer count = 20 case, do you know why is it so? Same question for the metric in Different Pserver Count, V2 is much better than Fluid.

No, need to figure it out.

[helinwang]

Thanks @typhoonzero !
"Currently multicore uses MKL threading which is much like OMP, I'd prefer to test with parallel_do support" -- do you mean "1 core" is just the setting for intel MKL, in terms of the available CPU resource for each trainer process is still the number of core of "Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz"?

Btw, can you take a look at the first question? (Does Single Node Single Thread mean single node, if so, the PServer Count: 10, Trainer Count: 20 below better be removed, since it's confusing.)

Thanks!

[typhoonzero]

"1 core" is just the setting for intel MKL, in terms of the available CPU resource for each trainer process is still the number of core of "Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz"?

Yes and No. The number of cores is determined by Kubernetes resource limits, and since I didn't use parallel_do in the test case, and gRPC uses async mode, this case does not utilize multi-core power at all.

I'll update the README.md today, thanks for pointing that out.