How to implement DataParallelEngine

[QiJune]

We should support a Net running on multi-GPUs. And users can just define a Net and set GPU ids, and the parallel running on multi-GPUs will be automatic.

In caffe2, NCCL and gloo are used to support multi-GPUs on multi-Servers. And both the operations in NCCL and gloo are represented as Operator.

In paddle now, we have implemented MultiGradientMachine and pserver. We might use NCCL to merge gradient in multi-GPUs in our new version. And should we take NCCL operations as Operator?

If NCCL operation is Operator, then one Net might corresponds to multi-GPUs. Or, just we take NCCL operation as a function, then we will have one Net corresponds to one GPU.

[Superjomn]

Had a short discussion with @QiJune a little earlier, get some preliminary ideas for multi-device executor:

• Maintain a Scope(named global_scope for example) in CPU, for merging gradients from multiple GPU cards.
• Create a Scope(named as child_scope_i) for each GPU card whose device id is i, set their father scope as global_scope.
• For each GPU card, create a new OpContext(named as ctx_i) to store streaming of this card
• Before Net runs, copy latest variables from global_scope to child_scope_i.
• Each GPU card will have a thread which executes Net.Run(child_scope_i, ctx_i)
• After all the cards' threads finished, merge all the gradients of child_scope_is and update global_scope.

Currently, Net is designed without the ownership of any computation resources (such as OpContexts) or memory resources (such as Variables or Scopes), so it is natural to have only one Net and run Net.Run with different Scopes and OpContexts in different devices.