Optimizer Design and Implementation

[jacquesqiao]

Goal

Implement Optimizer with new CompileTime & RunTime architecture.

optimizer used in book chapter:

1. Momentum.
   01.fit-a-line
   02.recognize_digits
   03.image_classification
   07.label_semantic_roles
2. AdaGrad
3. Adam
4. L2Regularization
5. ModelAverage

related work

Optimizer Framework

• Optimizer survey @jacquesqiao (Optimizer Survey #3672)
• optimizer Python API Design @jacquesqiao (Optimizer Design #4656)
• expose backward of ProgramDesc to python. @jacquesqiao (expose AppendBackward of ProgramDesc to python #4699)
• improve backward @jacquesqiao (create grad_var when run Backward pass #4796)
• create grad_var when run Backward pass @jacquesqiao (create grad_var when run Backward pass #4796)
• Python cpp sync @jacquesqiao (Review operators required by books. #4786)
• implement the interface of optimizer. @jacquesqiao (Impl optimizer #4734)

Optimize operators:

• Adagrad @kexinzhao (Implementing the Adagrad optimizer step operator #4558)
• Decayed Adagrad @kexinzhao (Implementing the Decayed Adagrad optimizer operator #4645)
• Adadelta @abhinavarora (Adding Adadelta optimization operator #4576)
• Adamax @abhinavarora (Implementing the Adamax optimizer operator #4538)
• AdamOp @abhinavarora (Adding the Adam Optimizer operator #4733)

Python Optimizer Wrapper

• Fix initialization of tensors (Fix tensor initialization in Optimizers Python wrappers #4975). Dependent on Initialization design doc #4852 @jacquesqiao Optimizer use init program #5275
• Momentum Op Wrapper @abhinavarora (Write Momentum Op Python Wrapper #4898)
• Adam Op Wrapper @abhinavarora Adding python wrapper for adam operator #5021
• Adagrad Op Wrapper @kexinzhao (Write Adagrad Op Python Wrapper #4916)
• Decayed Adagrad Op Wrapper @kexinzhao (Write Decayed Adagrad Op Python Wrapper #4993)
• Adamax Op Wrapper @abhinavarora Implementing the python wrapper for Adamax optimizer #5061
• Support Nesterov momentum in momentum wrapper @abhinavarora Adding nesterov momentum to python momentum wrapper #5055
• Increment global step if provided @abhinavarora Handling global step increment in optimizer python wrapper #5097

Related ToDos

• Implement the increment op for global step @abhinavarora(Implement increment op fo optimizer global step #4912)
• Fix Adam Op @abhinavarora (Fix Adam Op to remove update of Beta1 and Beta2 power accumulators outside the op #4909)
• Fix Adamax Op @abhinavarora (Fix Adamax Op to remove update of Beta1 Power Accumulator #4920)