RectifiedAdam: Store 'total_steps' hyperparameter as float

[olesalscheider]

Description

When restoring the RectifiedAdam optimizer from a checkpoint, restoring fails with the following error message:
ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: <tf.Tensor: shape=(), dtype=int32, numpy=0>

The reason is that RectifiedAdam stores total_steps as an integer hyper-parameter:
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/rectified_adam.py#L134

OptimizerV2 however creates variables by calling add_weight without a dtype in _create_hypers. In other words, it assumes that all hyper-parameters are floating point parameters.

Fixes #2361

Type of change

• Bug fix
• New Tutorial
• Updated or additional documentation
• Additional Testing
• New Activation and the changes conform to the activation contribution guidelines
• New Callback and the changes conform to the callback contribution guidelines
• New Image addition and the changes conform to the image op contribution guidelines
• New Layer and the changes conform to the layer contribution guidelines
• New Loss and the changes conform to the loss contribution guidelines
• New Metric and the changes conform to the metric contribution guidelines
• New Optimizer and the changes conform to the optimizer contribution guidelines
• New RNN Cell and the changes conform to the rnn contribution guidelines
• New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
• New Text addition and the changes conform to the text op contribution guidelines

Checklist:

• I've properly formatted my code according to the guidelines
  • By running Black + Flake8
  • By running pre-commit hooks
• This PR addresses an already submitted issue for TensorFlow Addons
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• This PR contains modifications to C++ custom-ops

How Has This Been Tested?

I have trained a neural network with this optimizer and resumed training in between. Without this change, restoring fails with the error message from above. With this change, it works as intended.

[bot-of-gabrieldemarmiesse]

@CyberZHG

You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

[olesalscheider]

The tests fail because total_steps is int in the constructor. I can fix that, but I am really not sure if this is the right way to handle this, or if this is unintended behavior from tensorflow.

[WindQAQ]

Thanks! Can you add a test that failed before the fix?

[bhack]

It is a little bit confusing as a float.

[olesalscheider]

It is a little bit confusing as a float.

That's what I thought. But if the constructor takes an int but stores it as a float internally, the tests fail because the optimizer can't be restored.

[google-cla]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[google-cla]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

[bhack]

@googlebot I consent