Optimizer Survey

[jacquesqiao]

fix #3655

Add some survey of optimizer in tensorflow, caffe2 and pytorch

after the refactoring, all the computation of paddle will be described by operator, optimizer will also be designed to use operator.

[jacquesqiao]

TensorFlow

The class Optimizer defines the API to add Ops to a graph. It adds two kinds of Operators:

1. Gradient Ops. Used to calculate the gradients for parameters.
2. Optimize Ops. Used to apply gradient to parameters.

在tf的optimier中对应的定义了两个接口compute_gradients和apply_gradients来实现上述的两个功能。同时为了方便使用，定义了一个minimize函数，将上述两个接口组合在一起。接口实现见下面的代码：

class Opetimzer(object):
	def compute_gradients(loss, var_list):
		"""
		Compute gradients of `loss` for the variables in `var_list`.
		
		This is the first part of `minimize()`. It returns a list
		of (gradient, variable) pairs where "gradient" is the gradient
		for "variable".
		"""
		...
		return grads_and_vars

	def apply_gradients(grads_and_vars):
		"""
		Apply gradients to variables.
		
		This is the second part of `minimize()`. It returns an 
		`Operation` that applies gradients.
		"""
		...
		return update_op

	def minimize(loss, var_list):
		"""Add operations to minimize `loss` by updating `var_list`.
		
		This method simply combines calls `compute_gradients()` and
		`apply_gradients()`.
		"""
		grads_and_vars = compute_gradients(loss, var_list)
		update_op = apply_gradients(grads_and_vars)
		return update_op

optimizer的使用

在tf中有两种使用Optimizer的方式：

1. Call minimize(loss, var_list) and run it's result

# 1. Create an optimizer with the desired parameters.
opt = GradientDescentOptimizer(learning_rate=0.1)

# 2. Add Ops to the graph to minimize a cost by updating a list of variables.
opt_op = opt.minimize(cost, var_list=<list of variables>)

# 3. Execute opt_op to do one step of training:
opt_op.run()

如果var_list不填的话，默认找到所有可以训练的parameter作为var_list。

minimize的返回值是一个operator，这个operator可以直接run，

In the training program you will just have to run the returned Op.

Demo code:

# Create the model
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)

# Define loss and optimizer
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# Train
tf.initialize_all_variables().run()
for i in range(1000):
  batch_xs, batch_ys = mnist.train.next_batch(100)
  train_step.run({x: batch_xs, y_: batch_ys})

2. Processing gradients before applying them.

Calling minimize() takes care of both computing the gradients and
applying them to the variables. If you want to process the gradients
before applying them you can instead use the optimizer in three steps:

1. Compute the gradients with compute_gradients().
2. Process the gradients as you wish.
3. Apply the processed gradients with apply_gradients().

Example:

# Create an optimizer.
opt = GradientDescentOptimizer(learning_rate=0.1)

# Compute the gradients for a list of variables.
grads_and_vars = opt.compute_gradients(loss, <list of variables>)

# grads_and_vars is a list of tuples (gradient, variable).  Do whatever you
# need to the 'gradient' part, for example cap them, etc.
capped_grads_and_vars = [(MyCapper(gv[0]), gv[1]) for gv in grads_and_vars]

# Ask the optimizer to apply the capped gradients.
opt.apply_gradients(capped_grads_and_vars)

这种方式主要用在多设备训练的时候，各个设备分别算出彼此的gradient，然后中间需要做一下merge，merge之后再update参数。具体例子可以参考：
https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py#L194

[jacquesqiao]

caffe2

caffe2的把添加gradient op和添加opitmize op分成了两个部分:

• gradient op由model负责添加
• optimize op 由optimizer负责添加。

optimizer构造出来之后，作为一个functor使用，每次调用都需要将net和需要优化的parameter和gradient的名字传给它，operator负责修改net，将对应的optimize op添加到net中去，添加完之后，就没有他什么事儿了，model负责在合适的时候调用相关的optimize op来进行参数更新。

下面写了一段代码来详细解释这个过程

from caffe2.python import brew, core, optimizer
from caffe2.python.model_helper import ModelHelper

model = ModelHelper(name="test")
fc1 = brew.fc(model, 'data', 'fc1', 100, 50)
fc2 = brew.fc(model, fc1, 'fc2', 50, 25)
pred = brew.fc(model, fc2, 'fc3', 25, 10)
(softmax, loss) = model.SoftmaxWithLoss(
    [pred, 'label'],
    ['softmax', 'loss'],
)
model.AddGradientOperators([loss])

sgd_1 = optimizer.SgdOptimizer(base_learning_rate=0.1)
sgd_2 = optimizer.SgdOptimizer(base_learning_rate=0.2)
adagrad = optimizer.AdagradOptimizer()

# Check same optimizer share the same learning rate.
sgd_1(model.net, model.param_init_net, "fc1_w", "fc1_w_grad")
sgd_1(model.net, model.param_init_net, "fc1_b", "fc1_b_grad")

sgd_2(model.net, model.param_init_net, "fc2_w", "fc2_w_grad")
sgd_2(model.net, model.param_init_net, "fc2_b", "fc2_b_grad")

adagrad(model.net, model.param_init_net, "fc3_w", "fc3_w_grad")
adagrad(model.net, model.param_init_net, "fc3_b", "fc3_b_grad")

print(model.net.Proto())

[jacquesqiao]

pytorch

pytorch是一个动态网络，完全用python实现的，而且是单机执行，实现的更加简单粗暴一些。

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)
for data in data_source():
    optimizer.zero_grad()
    output = model(data)
    loss = F.nll_loss(output, target)
    loss.backward()
    optimizer.step()

Pytorch的Optimizer在实现的时候需要把所有的参数传递进去，Pytorch的Parameter中包含了gradient的值，如上述代码所示，每次掉完 backward()之后会算出当前gradient的值，然后调用 optimizer.step.

optimizer.step内部有个for循环，依次按照策略更新每个Parameter的值。