design doc for implementation parameters in CPP.

[reyoung]

The step by step approach for how to implement Parameters in CPP

Here may be better to review.

[wangkuiyi]

CPP => C++

[wangkuiyi]

The title should be

# Design Doc: The C++ Class `Parameters`

[reyoung]

Done.

[wangkuiyi]

Is it that GradientMachine and NeuralNetwork does single-thread training, and MultiGradientMachine does concurrent training? If so, it seems that it is the responsibility of MultiGradientMachine, other than Parameters, to sync up among threads.

如果 GradientMachine, NeuralNetwork, MultiGradientMachine 都用到 Parameters，但是只有 MultiGradientMachine做并发训练，前两个classes都不做，那么对多线程的支持应该在 MultiGradientMachine 里面，而不应该在 Parameters 里面。

[reyoung]

MultiGradientMachine can support only one topology while training. But Parameters may be shared by many topologies. I think MultiGradientMachine should invoke Parameters.exchangeToMultiGPU(used_parameter_names) method when MultiGradientMachine is used only a subset of Parameters. Or let another class to do exchange job, such as exchanger = new ParameterExchanger(parameters, used_parameter_names), exchanger.exchange();

Another reason I want to extract Parameter Exchange/Gather logic from MultiGradientMachine is the MultiGradientMachine is a super class. It mixes the Multi-Devices computing logic, Parameter Exchange/Gather logic, synchronization together in a Single Class. It should be better and clearer that we extract some logic.

原因有如下几点:
1、MultiGradientMachine只处理了训练一个拓扑结构的情况，而Parameters可能在训练中被多个拓扑结构共享。于是多卡参数交换就和单个拓扑结构情况下不同(不是所有参数都要进行交换，而是选择某些参数进行交换)。当然，还是应该由MultiGradientMachine调用Parameters.exchange进行交换。
可能的实现手法是:

// ParameterExchanger负责参数交换的全部逻辑
auto exchanger = new ParameterExchanger(parameters, used_parameter_names);
exchanger.exchange();

2、另一个想要把参数交换逻辑提取出来的原因是，MultiGradientMachine是一个非常重的类，揉和了多个功能。例如多设备的计算，参数聚合分发，同步逻辑等等。如果我们在写Parameters的时候，把参数聚合逻辑分解出来，会让代码逻辑变得更清晰。

[reyoung]

Maybe global function is better.

[reyoung]

Done. Added this part into design doc.