Multi-GPU, multi-node development milestones.

[helinwang]

1. Single node multiple CPU threads
   1. Executor support multiple threads: @helinwang propose to drive this.
   2. Transpiling: convert user's input ProgramDesc to a ExecutionPlan that supports CPU-only sync-SGD data parallelism: @Yancey1989 .
2. Single node multiple GPUs
   1. Transpiling: convert user's input ProgramDesc to a ExecutionPlan that supports GPU sync-SGD data parallelism.
3. Multiple nodes
   1. operators for feeding data
   2. Transpiling: convert user's input ProgramDesc to a ExecutionPlan that runs on multiple nodes.
   3. Send / Recv OP
   4. ExecutionPlan partition: partition the single ExecutionPlan to multiple ExecutionPlans, each partitioned ExecutionPlan runs on a node. Send / Recv OP is added between edges that cross nodes.
4. Fault tolerant: single node failure stops the training job and causes a job restart.
   1. every executor should save state automatically and loads state upon restart.
5. Elastic ML: number of nodes can change without interrupting the training (training job will not stop).

Please comment if you have question or suggestions, thanks!

[typhoonzero]

Are these roadmaps in the order of what we are going to develop?

Append one TODO:

• operators for feeding data

[Yancey0623]

Some disscuess:

• Shall we also need a master role to dispatch training data? If so, how about implement the master as an op?
• I saw @typhoonzero implement the queue in send/recv op, shall we need to independed inqueue/dequeue op?

[helinwang]

@typhoonzero

Are these roadmaps in the order of what we are going to develop?

Yes, to my best understanding this is one viable order.

Append one TODO:

Thanks! Added to multiple node milestone.

@Yancey1989

Shall we also need a master role to dispatch training data? If so, how about implement the master as an op?

I think the master node is useful for monitoring the status of training data, but that does not seems to be a high priority since most of the researchers monitors the cost curve rather than the training data progress. Maybe we can postpone it.

I saw @typhoonzero implement the queue in send/recv op, shall we need to independed inqueue/dequeue op?

That's a great idea, I think so, since many other places needs queue too (e.g., read data OP output to a buffered queue).

[typhoonzero]

About placement design,

1. Represent placement using a string like "/pserver/CPU", "/trainer/GPU:0-7"
2. Using ProgramDesc and placement to form a ExecutionDesc?

[typhoonzero]

Whether ExecutionDesc is needed, to be discussed.

[helinwang]

This issue is out-of-date, closing.