Optimizer C_API

[dzhwinter]

上次讨论提到

Model Optimization Using Gradients

There are two ways to perform model optimization using gradients:

• On Client
  The client does multiple steps of forward and backward update. In each step, the gradients are >calculated and a new model is generated. After some steps, the client will calculate the difference >between the newest model and the old model at step 0. The difference will be updated to >parameter servers. Parameter servers will just update parameters using the difference without any >optimization using gradients (such as Adam and L1 regularization).
• On Parameter Server
  The client will send accumulated gradients to parameter servers, the parameter server will do the >optimization using gradients.

这两种更新参数的方法计划都支持。目前v1版本只支持1(On Client的方法)，由于两种方法都需要Optimizer更新的策略，因此选择将Optimizer封装成一个库。

ParameterServer为Go语言实现，需要一个Optimizer的C接口，定义如下

    // support data type same with @helin's client design doc, 
    typedef enum {
      PADDLE_ELEMENT_TYPE_INT32   = 0,
      PADDLE_ELEMENT_TYPE_UINT32  = 1,
      PADDLE_ELEMENT_TYPE_INT64   = 2,
      PADDLE_ELEMENT_TYPE_UINT64  = 3,
      PADDLE_ELEMENT_TYPE_FLOAT32 = 4,
      PADDLE_ELEMENT_TYPE_FLOAT64 = 5,
    } paddle_element_type;
    
    
    
    /*
    @brief update interface of optimizer, which will be used in 
    Trainer process
    ParameterServer process to support On ParameterServer optimize
    @param buffer : array of parameters
    @param datatype : datatype of parameter and gradient
    @param optimizer_name: optimizer_name as algorithm id, "SGD, Adam"
    @param gradient : array of gradients, which will be apply to parameters
    */
    void updateParameter(void *buffer, paddle_element_type datatype, const char* optimizer_name, const void* gradient);

1、是否能将sparseUpdate和denseUpdate都使用这一个接口？

SparseUpdate存储为SparseRowMatrix，可以复用这个接口。

2、是否能将Regularizer一起封装在这个库里？

On Client的参数更新方式已经和通信耦合，特别是SparseUpdate时候，由于Update的过程是lazy的，本地迭代了多次，Regularizer需要保存计算的轮数，并且需要在某次读取时候触发更新。没想到合适的办法拆分通信状态

Optimizer计划封装math库里的底层applySGD等操作，详细代码位置见：

https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/math/TrainingAlgorithmOp.cu#L25

后期接入Majel可以迁移这部分代码

[dzhwinter]

related design doc https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/cluster_train/pserver_client.md

[dzhwinter]

需要注意Regularizer的部分还是必须独立出来，目前代码中这个部分尝试optimizer和regularizer结合起来。看起来处于无用状态。
https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/parameter/OptimizerWithRegularizer.h#L63

希望的达到状态是：统一的库接管trainer和ParameterServer在优化方面的计算，封装一层math里的applySGD函数。

[helinwang]

const char* optimizer_name是不是改成enum更好一些？

[dzhwinter]

👍 放在一个proto文件定义这些声明？

enum OptimizerIdentifier {
  SGD =  0,
  ASGD = 1,
  ADAM = 2,
  // ...
};
    
    /*
    @brief update interface of optimizer, which will be used in 
    Trainer process
    ParameterServer process to support On ParameterServer optimize
    @param buffer : array of parameters
    @param datatype : datatype of parameter and gradient
    @param optimizer_name: optimizer_name as algorithm id, "SGD, Adam"
    @param gradient : array of gradients, which will be apply to parameters
    */
    void updateParameter(void *buffer, const void* gradient, OptimizerIdentifier optimizer, 
 paddle_element_type datatype);

这个接口非常简单，
确认两点，一个接口确实已经可以完全表达所有优化函数接口。
另外一点是regularizer确实做不到也作为一个函数接口提供给ParameterServer和Trainer分别调用。

[helinwang]

Optimizer计划封装math库里的底层applySGD等操作，详细代码位置见：
https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/math/TrainingAlgorithmOp.cu#L25

我看到引用的代码是gpu代码，提醒一下，现在pserver不需要支持GPU，太费GPU资源了，CPU也不一定慢很多。

建议Optimizer C++代码我们自己先写一份吧，现在Tensor也在重构，到时候需要的话再换到共享的Tensor库。

[helinwang]

请看https://github.com/PaddlePaddle/Paddle/blob/develop/doc/design/cluster_train/pserver_client.md 其中的

int paddle_begin_init_params(paddle_pserver_client* client, const char* config_proto);

是需要一个proto文件定义pserver的设置。但是建议Optimizer不用去理解proto，由pserver理解完了proto再根据proto要求创建Optimizer。

[helinwang]

这个接口非常简单，
确认两点，一个接口确实已经可以完全表达所有优化函数接口。
另外一点是regularizer确实做不到也作为一个函数接口提供给ParameterServer和Trainer分别调用。

"一个接口确实已经可以完全表达所有优化函数接口。"：应该是可以了，可能需要稍作修改：

typedef enum {
  SGD =  0,
  ASGD = 1,
  ADAM = 2,
  // ...
} optimizer_identifier;
typdef struct optimizer optimizer; // forward declaration
optimizer* paddle_create_optimizer(optimizer_identifier identifier);
void paddle_release_optimizer(optimizer* o);
int paddle_update_parameter(optimizer* o, void *buffer, paddle_element_type datatype, const void* gradient, int num_bytes, double learning_rate);

添加num_bytes，作为buffer和gradient的长度。
添加learning_rate。
我把OptimizerIdentifier改成了optimizer_identifier，updateParameter改成了paddle_update_parameter，貌似咱们的c代码用的是下划线风格的？(https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/capi/matrix.h#L30)
另外c没有namespace，一般需要用个前缀作为namespace。所以我改成了paddle_update_parameter。

[dzhwinter]

赞同comment，开始着手重写一份optimizer

[dzhwinter]

@helinwang
I found a new problem. How do we support different CPU instructions?
for example,
I had implemented the GradientDescent optimizer, but I found that old Paddle support avx through a difference interface. Do we need to add these feature in the new library?

Paddle/paddle/parameter/ParameterUpdateFunctions.h

Line 48 in b0305af

void sgdUpdateAvx(float learningRate,

[helinwang]

@dzhwinter avx is supported in almost all modern CPUs. However, maybe let's do non-avx CPU first (development time is faster as well because of no need to learn the avx intrinsics) and do avx if optimization is necessary.

[dzhwinter]

We had already done the SGD algorithm job in v1, no more effort should be paid here since we are in a hurry.
But we need to re-think about the create_optimizer interface. Please take a look at this old optimizer configuration.
https://github.com/PaddlePaddle/Paddle/blob/develop/proto/TrainerConfig.proto#L21

I found that parse the configuration is inevitable.
for example,

// Adagrad optimizer
  optional double ada_epsilon = 24 [default = 1e-6];
  optional double ada_rou = 26 [default = 0.95];

  // Options Adam Optimizer //
  ////////////////////////////
  optional double adam_beta1 = 33 [default = 0.9];
  optional double adam_beta2 = 34 [default = 0.999];
  optional double adam_epsilon = 35 [default = 1e-8];

How about pass a serialized proto file to describe the configuration? we need this config when we create optimizer, otherwise we need a standalone create function in pure C for every optimizer method.

[helinwang]

@dzhwinter Before we wanted the optimizer to be super simple, as few state as possible, and the Go part do the messy part like manage momentum memory, parsing config, so Go store many states (like momentum memory, since it needs to parse the config to know if it's momentum based optimizer).

If we want to let optimizer parse and create the specific optimizer, Go better don't parse it again. So all the config related states will need to be managed by C++, that would complicate interfaces (e.g., C++ will need to manage momentum memory for each parameter, so it needs the parameter name exposed by the C++-Go API).

I am a little worried about it complicating things more than "need a standalone create a function in pure C for every optimizer method.". How do you think?

[dzhwinter]

I have no tendency if the Go part should responsible with the states and parsing config.

Actually, I just thought that if we write in Go part way, we need a bunch of different create function, update function
by the way, I cannot agree anymore that we only put pass stateless code into optimizer and do not need the oprimizer understanding config. Firstly, if we treat optimizer as a library, we should not manage a state. Secondly, understanding config and do config checking should be happen in unique place , which is the trainer in Go part . We suffering the config parsing problem for a long time.
I just think that maybe there is someway that we can make it better?

[dzhwinter]

I just write some pseudo code here show what will happen if we put it in Go part.

// for create optimizer function signature
create_SGDOptimizer(learning_rate)
create_Momentum(learning_rate, mu)
create_Adam(learning_rate, )
// for update function signature

updateSGD(parameter,  gradient, mu, learning_rate);

updateMomentum(parameter, momentum,  gradient, mu, learning_rate);

updateAdam(parameter, momentum,  gradient, mu, rha, learning_rate);