Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tfp.optimizer.lbfgs_minimize as Keras optimizer? #565

Open
Hoeze opened this issue Sep 19, 2019 · 5 comments
Open

tfp.optimizer.lbfgs_minimize as Keras optimizer? #565

Hoeze opened this issue Sep 19, 2019 · 5 comments

Comments

@Hoeze
Copy link

Hoeze commented Sep 19, 2019

Hi, is there a way to use tfp.optimizer.lbfgs_minimize as Keras optimizer?
This would be quite useful in certain cases where the loss function is approximately quadratic.

A colleague of mine would very much need it since an autoencoder written in R with negative-binomial loss converges faster than its Keras counterpart.

@dynamicwebpaige
Copy link
Contributor

Thanks for the recommendation, @Hoeze! Are you using stand-alone Keras for your work, or tf.keras?

@Hoeze
Copy link
Author

Hoeze commented Nov 9, 2019

@dynamicwebpaige Thanks for your answer.
We are usually using the default Keras API with Tensorflow as backend.
For special needs, we construct Keras layers with the Tensorflow API.

I would be cool if it was implemented as a Tensorflow optimizer similar to ADAM:
https://github.com/tensorflow/tensorflow/blob/r2.1/tensorflow/python/training/adam.py
This way it would be more easy to use in plain Tensorflow as well, since switching to L-BFGS would only incorporate changing a single line of code.

The big problem with tfp.optimizer.lbfgs_minimize was that we had to provide it with a function returning the loss. Then it does some magic and eventually the result has converged. One cannot record the single optimization steps.
I am not sure how this black-box optimization fits with Keras, since e.g. the EarlyStopping callback would not work this way.

@dario-passos
Copy link

dario-passos commented Jul 6, 2020

@Hoeze I was looking for the same functionality and I found this blog that show how to use the lbfgs_minimize() with a tf.keras model: https://pychao.com/2019/11/02/optimize-tensorflow-keras-models-with-l-bfgs-from-tensorflow-probability/

However it would be very useful if the tf.keras development team could implement this as a Tensorflow optimizer in a future update, just like you suggest!!
Cheers

@bhack
Copy link

bhack commented Apr 14, 2021

You can follow also tensorflow/tensorflow#48167

@brianwa84
Copy link
Contributor

That blog post is a great resource for how to glue the functional form in TFP together with tf.Variables (keras). I think we should probably fix the requirement that the parameters be a single 1D tensor, as this is inconsistent with other places in TFP, but it is not currently a high priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants