Refine code(momentum_op, rmsprop_op )

[chengduoZH]

fix #6002

[qingqing01]

Whether the framework::EigenScalar can be used for lr instead of broadcast ?

[chengduoZH]

I've tried it, but the result is wrong.

[dzhwinter]

This implement is almost same.
it's better to remove the v_out.device

[chengduoZH]

Great, I think it's better to remove the v_out.device, too.

[chengduoZH]

I find that v_out and v point to the same memory, this is to say the writer use inplace method in python interface. And it is necessary to update v_out in momentum_op, so v_out.device can't be removed.

[chengduoZH]

I find that the implementation of momentum_op seems to be wrong.
Document description:

velocity = mu * velocity + gradient 
if (use\_nesterov):   
  param = param - gradient * learning\_rate + mu * velocity * learning\_rate 
else:   
  param = param - learning\_rate * velocity. 

Code implementation:

Paddle/paddle/operators/momentum_op.h

Lines 51 to 57 in 728e8b1

	v_out.device(place) = v * mu + g;
	if (use_nesterov) {
	p_out.device(place) = p - g * lr.broadcast(grad_dsize) +
	v_out * mu * lr.broadcast(grad_dsize);
	} else {
	p_out.device(place) = p - lr.broadcast(grad_dsize) * v_out;
	}

[chengduoZH]

The writer use inplace method in python interface, this is to say that v_out and v point to the same memory. I think this should be annotated in the code.

Paddle/python/paddle/v2/fluid/optimizer.py

Lines 265 to 278 in 21053c1

	momentum_op = block.append_op(
	type=self.type,
	inputs={
	"Param": param_and_grad[0],
	"Grad": param_and_grad[1],
	"Velocity": velocity_acc,
	"LearningRate": self._create_param_lr(param_and_grad)
	},
	outputs={
	"ParamOut": param_and_grad[0],
	"VelocityOut": velocity_acc
	},
	attrs={"mu": self._momentum,
	"use_nesterov": self._use_nesterov})