Skip to content

Adding the Adam Optimizer operator #4733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Oct 12, 2017
Merged

Adding the Adam Optimizer operator #4733

merged 8 commits into from
Oct 12, 2017

Conversation

abhinavarora
Copy link
Contributor

Fixes #4377

Yang and others added 5 commits September 26, 2017 13:34
add adam op
749183d 
moment1_out = beta1 * moment1 + (1 − beta1) * grad
moment2_out = beta2 * moment2 + (1 − beta2) * grad * grad
moment1_hat =  moment1_out / (1 - beta1^t)
moment2_hat =  moment2_out / (1 - beta2^t)
param_out = param - learning_rate * moment1_hat / (sqrt(moment2_hat) +
epsilon)
fix moment 2
bdaefdc
Merge remote-tracking branch 'origin/develop' into adam_op
f638506
Adding the Adam optimization operator
6e772ef
Adding more tests for Adam op
cebb7dc
@abhinavarora
Copy link
Contributor Author

Adam Updates:

moment1_out = beta1 * moment1 + (1 − beta1) * grad
moment2_out = beta2 * moment2 + (1 − beta2) * grad * grad
beta1_pow_out = beta1_pow * beta1
beta2_pow_out = beta2_pow * beta2
learning_rate_t = learning_rate_t *
                  sqrt(1 - beta2_pow_out) / (1 - beta1_pow_out)
param_out = param - learning_rate_t * moment1/ (sqrt(moment2) + epsilon)

@abhinavarora abhinavarora self-assigned this Oct 12, 2017
tonyyang-svail
tonyyang-svail approved these changes Oct 12, 2017
View reviewed changes
Copy link

@tonyyang-svail tonyyang-svail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good. Great work!

@abhinavarora abhinavarora merged commit 1168003 into PaddlePaddle:develop Oct 12, 2017
@abhinavarora abhinavarora deleted the adam_op branch October 12, 2017 20:36
@jacquesqiao jacquesqiao mentioned this pull request Oct 12, 2017
Closed
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonyyang-svail tonyyang-svail tonyyang-svail approved these changes

Assignees

@abhinavarora abhinavarora

Labels
OpPorting
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@abhinavarora @tonyyang-svail @qingqing01