Move regularization and error clipping to parameter server side.

[typhoonzero]

Move regularization and error clipping to parameter server side in the distributed transpiler.

[helinwang]

• I think if the normalization is done on the trainer side, making it part of the loss function, then the gradient update (parameter server's job) don't need to know about the normalization, so the update is always element wise, thus we don't need to do anything special when each parameter is sharded into multiple parameter servers. I briefly looked into the code few days ago, it seems that the normalization is already part of the loss function (@lcy-seso if you happen to be familiar with the L1/L2 normalization implementation, could you confirm? Thank you! If not don't worry about it, I will double check). I prefer that we can treat normalization (e.g., L1, L2) as part of the loss function (most general).

• Gradient clipping we can do on the parameter server side, it seems not very hard.

[typhoonzero]

Sorry, my fault, I mean regularization and clipping. See: https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/v2/fluid/optimizer.py#L203

Thanks very much for your comment.

[helinwang]

Thanks, very important information.

[typhoonzero]

I think the simple version could put decay and clipping to server side should do the basic work.