[BREAKING CHANGE] deel-lip upgrade to Keras 3.0 #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In Keras 3, layers and models do not have anymore `input_shape` information. It is now retrieved through `layer.input.shape`. Note that `layer.input` can be a list of KerasTensor (e.g. in Add/Concatenate layers). It is thus necessary to iterate over `layer.input` list to get all shapes.
In Keras 3, Layer.add_weight() first argument is now `shape` instead of `name`. We use keyword argument to avoid future changes.
In Keras 3, tf.Variable.read_value() was removed. We replaced with `tf.Variable.value`.
In Keras 3, tf.keras.losses.Reduction API is not used anymore. Moreover, the reduction="auto" was removed. We now use strings and set values to the default `reduction="sum_over_batch_size"`.
In Keras 2, the order was: reduction, name Now in Keras 3, the order is: name, reduction We add keyword arguments to avoid confusion.
Before in Keras 2, a single scalar was accepted to define a 1-D input shape. Now in Keras 3, it must be a tuple of a single element.
Before in Keras 2, `model.save()` can support TF SavedModel format in a folder. Now in Keras 3, `model.save()` only accepts Keras format (`.keras` extension)
Before in Keras 2, `model.save(path)` created path if not exist. Now in Keras 3, it does not create the path. The solution is to makedirs in a setup function for unit tests.
In Keras 3, `lr` should be replaced with `learning_rate` argument.
In Keras 3, the `output_padding` argument in Conv2DTranspose was removed. In this commit, we simply remove all references to `output_padding`. In a later commit, our SpectralConv2DTranspose layer will be updated to reflect the new internal code of Conv2DTranspose Keras layer.
cofri
force-pushed
the
fix/upgrade_tf_2.17
branch
from
6f3bb28
to
5da6b92
Compare
In Keras 2, arguments of keras.Sequential() were "layers" and "name". In Keras 3, there is a new argument in second position: "layers", "trainable" and "name". To avoid confusion, we use keyword arguments.
TODO: MonitorCallback is based on TensorBoard to monitor variables. This must be handled differently (lazy import of TensorFlow, or use of tensorboard directly).
TODO: How to handle `@tf.function` with Keras 3? TODO: tf.float32.min seems to have no equivalent. Is it ok to use K.min(y_pred)?
TODO: How to handle @tf.function ? TODO: boolean_mask replaced with a[mask], ok?
In Keras 3, `swap_memory` and `parallel_iterations` are not arguments in while_loop(). In Keras 3, `l2_normalize()` does not exist. We recreate one in utils.py based on TensorFlow implementation. In Keras 3, `keras.ops.norm()` can only be applied on 1-D or 2-D tensors. In the power iteration method, we flatten `u` to compute the stopping condition.
In Keras 3, `keras.ops.matmul()` only takes tensors A and B, and no other arguments like `transpose_a` and `transpose_b`.
In Keras 3, l2_normalize operation does not exist. We recreate a Keras one based on TensorFlow implementation.
TODO: How to replace @tf.function? TODO: how to replace tf.TensorShape?
TODO: How to replace TensorShape?
TODO: How to replace @tf.function? TODO: OK to replace custom gradient for sqrt operation ? sqrt(x + eps) to avoid infinite gradient when x=0
TODO: @tf.function? TODO: @Property for kernel(self)
New argument "groups" is added in Conv2D-derived classes. Since Lipschitz convolutions do not support groups other than 1, we check the value in the __init__. `call` functions are updated based on the code of convolutional layers in Keras 3.
TODO: What to do with tensorboard log?
TODO: What to do with tensorboard?
TODO: tf.summary / Tensorboard ?
In Keras 3, when model.save() and load_model(), custom variables are not saved by default. It is necessary to override save_own_variables() and load_own_variables() to manually store custom variables. Note that Conv2DTranspose has not the same behaviour: all trainable and non-trainable variables are saved/loaded. It is thus not required to override these two methods.
Tutorial notebooks are updated to Keras 3 and new DEEL-LIP code.
Tox environments are cleaned up to focus on the supported versions.
cofri
force-pushed
the
fix/upgrade_tf_2.17
branch
from
3626290
to
62a9d2c
Compare
SVD operation in Keras 3 does not handle "compute_uv". Bug was fixed in Keras 3.6.0 (next version to be released). Waiting for this version, a hot fix is used where we compute u and v for nothing. This commit could be reverted when Keras 3.6 is released.
cofri
force-pushed
the
fix/upgrade_tf_2.17
branch
from
62a9d2c
to
ed92ee3
Compare
franckma31
approved these changes
Oct 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TensorFlow 2.16 version and above come with Keras 3: a lot of major changes and incompatibilities have been introduced with previous TF versions.
We then propose a new deel-lip version 2.0.0 compatible with TensorFlow 2.16+ and Keras 3.
This PR has two sets of commits:
tf.kerasfunctions and classes.