Pkgoogle/efficient net migration #1778
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattdangerw
reviewed
Aug 16, 2024
There was a problem hiding this comment.
Thanks for the PR!
- I think the first big question we should ask is if we want separate
EfficientNetV1andEfficientNetV2classes. Keeping the separate will be more work, but could make sense if the architectures are very diverged. - After that, it may make sense to land just one new public symbol at a time. Otherwise this could take a while to clean up and land.
- Let's make sure to rewrite for the "local style" of this repo. We are trying to make sure we aren't just doing a code dump, but actually come up with a more consistent and maintainable state than the keras-cv source.
| min_depth=8, | ||
| activation="swish", | ||
| input_shape=(None, None, 3), | ||
| input_tensor=None, |
| **kwargs, | ||
| ): | ||
| # Determine proper input shape | ||
| img_input = keras_utils.parse_model_inputs(input_shape, input_tensor) |
keras_nlp/src/utils/keras_utils.py
Outdated
| return keras.Model(inputs=model.inputs, outputs=outputs) | ||
|
|
||
|
|
||
| def detect_if_tensorflow_uses_keras_3(): |
There was a problem hiding this comment.
is any of this keras 3 stuff used? if so we should remove it for sure. the library is now fully Keras 3
|
|
||
| @keras_nlp_export("keras_nlp.models.EfficientNetV1Backbone") | ||
| @keras.saving.register_keras_serializable(package="keras_nlp.models") | ||
| class EfficientNetV1Backbone(Backbone): |
There was a problem hiding this comment.
Can you give a summary of how much is different between V1 and V2?
We can leave separate, but it will come with a cost for sure, of all the testing and tasks we need to add. Possibly high effort.
Supporting multi variants is not an issue, as long as we capture the difference in our config options. But it's unclear to me what the difference are.
There was a problem hiding this comment.
Sg, let me work on this first, as that might change the rest of this.
|
|
||
|
|
||
| @keras_nlp_export("keras_nlp.models.EfficientNetV1Backbone") | ||
| @keras.saving.register_keras_serializable(package="keras_nlp.models") |
There was a problem hiding this comment.
We should not need this line.
keras_nlp/src/layers/fusedmbconv.py
Outdated
| return x | ||
|
|
||
| def get_config(self): | ||
| config = { |
There was a problem hiding this comment.
| from keras_nlp.src.utils.keras_utils import get_feature_extractor | ||
|
|
||
|
|
||
| class EfficientNetV1BackboneTest(TestCase): |
There was a problem hiding this comment.
Follow the form of other backbone tests that have been added. We have some common test routines.
|
|
||
|
|
||
| @keras_nlp_export("keras_nlp.models.EfficientNetV2Backbone") | ||
| class EfficientNetV2Backbone(Backbone): |
There was a problem hiding this comment.
I think this should extend the FeaturePyramidBackbone added #1769, and will need to update to that usage I think.
keras_nlp/src/utils/keras_utils.py
Outdated
| @@ -128,3 +128,101 @@ def standardize_data_format(data_format): | |||
| f"Received: data_format={data_format}" | |||
| ) | |||
| return data_format | |||
|
|
|||
|
|
|||
| def get_feature_extractor(model, layer_names, output_keys=None): | |||
There was a problem hiding this comment.
Let's remove this if we can and try to write tests without it.
keras_nlp/src/utils/keras_utils.py
Outdated
| return input_tensor | ||
|
|
||
|
|
||
| def correct_pad_downsample(inputs, kernel_size): |
There was a problem hiding this comment.
Is this something we will need generally? Why haven't any of our other backbones yet. Let's just add it to efficientnet if we are unsure if it's general.
pkgoogle
force-pushed
the
pkgoogle/efficient_net_migration
branch
from
e2662b9 to
12c898e
Compare
pkgoogle
force-pushed
the
pkgoogle/efficient_net_migration
branch
from
12c898e to
f1dd1f8
Compare
mattdangerw
reviewed
Aug 23, 2024
| number of filters the rounding process potentially produces | ||
| (as per v1). | ||
| stem_conv_padding: str, can be 'same' or 'valid'. Padding for the stem. | ||
| bn_momentum: float, momentum for the moving average calcualtion in the |
There was a problem hiding this comment.
Unless we already have "bn" in a lot of public api, I would right this out more. "batch_norm_momentum"
| True, inputs will be passed through a `Rescaling(1/255.0)` layer. | ||
| width_coefficient: float, scaling coefficient for network width. | ||
| depth_coefficient: float, scaling coefficient for network depth. | ||
| dropout_rate: float, dropout rate before final classifier layer. |
There was a problem hiding this comment.
I think we just call this dropout in other backbones.
| width_coefficient: float, scaling coefficient for network width. | ||
| depth_coefficient: float, scaling coefficient for network depth. | ||
| dropout_rate: float, dropout rate before final classifier layer. | ||
| drop_connect_rate: float, dropout rate at skip connections. The default |
There was a problem hiding this comment.
consolidated all the arguments into just "dropout"
| MBConvBlock, but instead of using a depthwise convolution and a 1x1 | ||
| output convolution blocks fused blocks use a single 3x3 convolution | ||
| block. | ||
| skip_connection_dropout: float, dropout rate at skip connections. |
There was a problem hiding this comment.
i think the actual python and docstring are out of sync, also group all dropout params next to each other.
There was a problem hiding this comment.
consolidated all the arguments into just "dropout"
| stackwise_strides: list of ints, stackwise_strides for each conv block. | ||
| stackwise_squeeze_and_excite_ratios: list of ints, the squeeze and | ||
| excite ratios passed to the squeeze and excitation blocks. | ||
| stackwise_conv_types: list of strings. Each value is either 'v1', |
There was a problem hiding this comment.
we need to actually apply this per layer? not per backbone?
There was a problem hiding this comment.
also we called this block_type for resnet. Does that need to be different here?
There was a problem hiding this comment.
original EfficientNetV2 implementation had it per layer -- but I don't mind either way. Changed to "stackwise_block_types".
There was a problem hiding this comment.
per layer sounds good then!
| stackwise_expansion_ratios=[1, 4, 4, 4, 6, 6], | ||
| stackwise_squeeze_and_excite_ratios=[0.0, 0.0, 0, 0.25, 0.25, 0.25], | ||
| stackwise_strides=[1, 2, 2, 2, 1, 2], | ||
| stackwise_conv_types=[ |
There was a problem hiding this comment.
maybe ["fused"] * 3 + ["unfused"] * 3 just so this formats on one line?
| se_ratio=squeeze_and_excite_ratio, | ||
| activation=activation, | ||
| dropout_rate=drop_connect_rate * block_id / blocks, | ||
| name="block{}{}_".format(i + 1, letter_identifier), |
There was a problem hiding this comment.
looks the same in both case just move a name = ... line up near letter_identifier
There was a problem hiding this comment.
refactored to above and changed to f-string format.
|
|
||
| class FusedMBConvBlock(keras.layers.Layer): | ||
| """ | ||
| Implementation of the FusedMBConv block (Fused Mobile Inverted Residual |
There was a problem hiding this comment.
Keep the one liner on the first line and keep it short.
| **kwargs | ||
| ): | ||
| """ | ||
| Implementation of the MBConv block (Mobile Inverted Residual Bottleneck) |
There was a problem hiding this comment.
same, one liner on the quote line, and keep it short
|
@pkgoogle most of the test failures look easily fixable, convert list to tuple when comparing against a shape. |
mattdangerw
pushed a commit
to mattdangerw/keras-hub
that referenced
this pull request
Sep 10, 2024
* migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes
mattdangerw
pushed a commit
that referenced
this pull request
Sep 11, 2024
* migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes
mattdangerw
pushed a commit
that referenced
this pull request
Sep 13, 2024
* migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes
mattdangerw
pushed a commit
that referenced
this pull request
Sep 17, 2024
* migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes
divyashreepathihalli
added a commit
that referenced
this pull request
Sep 25, 2024
* Add VGG16 backbone (#1737) * Agg Vgg16 backbone * update names * update tests * update test * add image classifier * incorporate review comments * Update test case * update backbone test * add image classifier * classifier cleanup * code reformat * add vgg16 image classifier * make vgg generic * update doc string * update docstring * add classifier test * update tests * update docstring * address review comments * code reformat * update the configs * address review comments * fix task saved model test * update init * code reformatted * Add `ResNetBackbone` and `ResNetImageClassifier` (#1765) * Add ResNetV1 and ResNetV2 * Address comments * Add CSP DarkNet backbone and classifier (#1774) * Add CSP DarkNet * Add CSP DarkNet * snake_case function names * change use_depthwise to block_type * Add `FeaturePyramidBackbone` and port weights from `timm` for `ResNetBackbone` (#1769) * Add FeaturePyramidBackbone and update ResNetBackbone * Simplify the implementation * Fix CI * Make ResNetBackbone compatible with timm and add FeaturePyramidBackbone * Add conversion implementation * Update docstrings * Address comments * Add DenseNet (#1775) * Add DenseNet * fix testcase * address comments * nit * fix lint errors * move description * Add ViTDetBackbone (#1776) * add vit det vit_det_backbone * update docstring * code reformat * fix tests * address review comments * bump year on all files * address review comments * rename backbone * fix tests * change back to ViT * address review comments * update image shape * Add Mix transformer (#1780) * Add MixTransformer * fix testcase * test changes and comments * lint fix * update config list * modify testcase for 2 layers * update input_image_shape -> image_shape (#1785) * update input_image_shape -> image_shape * update docstring example * code reformat * update tests * Create __init__.py (#1788) add missing __init__ file to vit_det * Hack package build script to rename to keras-hub (#1793) This is a temporary way to test out the keras-hub branch. - Does a global rename of all symbols during package build. - Registers the "old" name on symbol export for saving compat. - Adds a github action to publish every commit to keras-hub as a new package. - Removes our descriptions on PyPI temporarily, until we want to message this more broadly. * Add CLIP and T5XXL for StableDiffusionV3 (#1790) * Add `CLIPTokenizer`, `T5XXLTokenizer`, `CLIPTextEncoder` and `T5XXLTextEncoder`. * Make CLIPTextEncoder as Backbone * Add `T5XXLPreprocessor` and remove `T5XXLTokenizer` Add `CLIPPreprocessor` * Use `tf = None` at the top * Replace manual implementation of `CLIPAttention` with `MultiHeadAttention` * Add Bounding Box Utils (#1791) * Bounding box utils * - Correct test cases * - Remove hard tensorflow dtype * - fix api gen * - Fix import for test cases - Use setup for converters test case * - fix api_gen issue * - FIx api gen * - Fix api gen error * - Correct test cases as per new api changes * mobilenet_v3 added in keras-nlp (#1782) * mobilenet_v3 added in keras-nlp * minor bug fixed in mobilenet_v3_backbone * formatting corrected * refactoring backbone * correct_pad_downsample method added * refactoring backbone * parameters updated * Testcaseupdated, expected output shape corrected * code formatted with black * testcase updated * refactoring and description added * comments updated * added mobilenet v1 and v2 * merge conflict resolved * version arg removed, and config options added * input_shape changed to image_shape in arg * config updated * input shape corrected * comments resolved * activation function format changed * minor bug fixed * minor bug fixed * added vision_backbone_test * channel_first bug resolved * channel_first cases working * comments resolved * formatting fixed * refactoring --------- Co-authored-by: ushareng <usha.rengaraju@gmail.com> * Pkgoogle/efficient net migration (#1778) * migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes * Add the ResNet_vd backbone (#1766) * Add ResNet_vd to ResNet backbone * Addressed requested parameter changes * Fixed tests and updated comments * Added new parameters to docstring * Add `VAEImageDecoder` for StableDiffusionV3 (#1796) * Add `VAEImageDecoder` for StableDiffusionV3 * Use `keras.Model` for `VAEImageDecoder` and follows the coding style in `VAEAttention` * Replace `Backbone` with `keras.Model` in `CLIPTextEncoder` and `T5XXLTextEncoder` (#1802) * Add pyramid output for densenet, cspDarknet (#1801) * add pyramid outputs * fix testcase * format fix * make common testcase for pyramid outputs * change default shape * simplify testcase * test case change and add channel axis * Add `MMDiT` for StableDiffusionV3 (#1806) * Add `MMDiT` * Update * Update * Update implementation * Add remaining bbox utils (#1804) * - Add formats, iou, utils for bounding box * - Add `AnchorGenerator`, `BoxMatcher` and `NonMaxSupression` layers * - Remove scope_name not required. * use default keras name scope * - Correct format error * - Remove layers as of now and keep them at model level till keras core supports them * - Correct api_gen * Fix timm conversion for rersnet (#1814) * Add `StableDiffusion3` * Fix `_normalize_inputs` * Separate CLIP encoders from SD3 backbone. * Simplify `text_to_image` function. * Address comments * Minor update and add docstrings. * Add VGG16 backbone (#1737) * Agg Vgg16 backbone * update names * update tests * update test * add image classifier * incorporate review comments * Update test case * update backbone test * add image classifier * classifier cleanup * code reformat * add vgg16 image classifier * make vgg generic * update doc string * update docstring * add classifier test * update tests * update docstring * address review comments * code reformat * update the configs * address review comments * fix task saved model test * update init * code reformatted * Add `ResNetBackbone` and `ResNetImageClassifier` (#1765) * Add ResNetV1 and ResNetV2 * Address comments * Add CSP DarkNet backbone and classifier (#1774) * Add CSP DarkNet * Add CSP DarkNet * snake_case function names * change use_depthwise to block_type * Add `FeaturePyramidBackbone` and port weights from `timm` for `ResNetBackbone` (#1769) * Add FeaturePyramidBackbone and update ResNetBackbone * Simplify the implementation * Fix CI * Make ResNetBackbone compatible with timm and add FeaturePyramidBackbone * Add conversion implementation * Update docstrings * Address comments * Add DenseNet (#1775) * Add DenseNet * fix testcase * address comments * nit * fix lint errors * move description * Add ViTDetBackbone (#1776) * add vit det vit_det_backbone * update docstring * code reformat * fix tests * address review comments * bump year on all files * address review comments * rename backbone * fix tests * change back to ViT * address review comments * update image shape * Add Mix transformer (#1780) * Add MixTransformer * fix testcase * test changes and comments * lint fix * update config list * modify testcase for 2 layers * update input_image_shape -> image_shape (#1785) * update input_image_shape -> image_shape * update docstring example * code reformat * update tests * Create __init__.py (#1788) add missing __init__ file to vit_det * Hack package build script to rename to keras-hub (#1793) This is a temporary way to test out the keras-hub branch. - Does a global rename of all symbols during package build. - Registers the "old" name on symbol export for saving compat. - Adds a github action to publish every commit to keras-hub as a new package. - Removes our descriptions on PyPI temporarily, until we want to message this more broadly. * Add CLIP and T5XXL for StableDiffusionV3 (#1790) * Add `CLIPTokenizer`, `T5XXLTokenizer`, `CLIPTextEncoder` and `T5XXLTextEncoder`. * Make CLIPTextEncoder as Backbone * Add `T5XXLPreprocessor` and remove `T5XXLTokenizer` Add `CLIPPreprocessor` * Use `tf = None` at the top * Replace manual implementation of `CLIPAttention` with `MultiHeadAttention` * Add Bounding Box Utils (#1791) * Bounding box utils * - Correct test cases * - Remove hard tensorflow dtype * - fix api gen * - Fix import for test cases - Use setup for converters test case * - fix api_gen issue * - FIx api gen * - Fix api gen error * - Correct test cases as per new api changes * mobilenet_v3 added in keras-nlp (#1782) * mobilenet_v3 added in keras-nlp * minor bug fixed in mobilenet_v3_backbone * formatting corrected * refactoring backbone * correct_pad_downsample method added * refactoring backbone * parameters updated * Testcaseupdated, expected output shape corrected * code formatted with black * testcase updated * refactoring and description added * comments updated * added mobilenet v1 and v2 * merge conflict resolved * version arg removed, and config options added * input_shape changed to image_shape in arg * config updated * input shape corrected * comments resolved * activation function format changed * minor bug fixed * minor bug fixed * added vision_backbone_test * channel_first bug resolved * channel_first cases working * comments resolved * formatting fixed * refactoring --------- Co-authored-by: ushareng <usha.rengaraju@gmail.com> * Pkgoogle/efficient net migration (#1778) * migrating efficientnet models to keras-hub * merging changes from other sources * autoformatting pass * initial consolidation of efficientnet_backbone * most updates and removing separate implementation * cleanup, autoformatting, keras generalization * removed layer examples outside of effiicient net * many, mainly documentation changes, small test fixes * Add the ResNet_vd backbone (#1766) * Add ResNet_vd to ResNet backbone * Addressed requested parameter changes * Fixed tests and updated comments * Added new parameters to docstring * Add `VAEImageDecoder` for StableDiffusionV3 (#1796) * Add `VAEImageDecoder` for StableDiffusionV3 * Use `keras.Model` for `VAEImageDecoder` and follows the coding style in `VAEAttention` * Replace `Backbone` with `keras.Model` in `CLIPTextEncoder` and `T5XXLTextEncoder` (#1802) * Add pyramid output for densenet, cspDarknet (#1801) * add pyramid outputs * fix testcase * format fix * make common testcase for pyramid outputs * change default shape * simplify testcase * test case change and add channel axis * Add `MMDiT` for StableDiffusionV3 (#1806) * Add `MMDiT` * Update * Update * Update implementation * Add remaining bbox utils (#1804) * - Add formats, iou, utils for bounding box * - Add `AnchorGenerator`, `BoxMatcher` and `NonMaxSupression` layers * - Remove scope_name not required. * use default keras name scope * - Correct format error * - Remove layers as of now and keep them at model level till keras core supports them * - Correct api_gen * Fix timm conversion for rersnet (#1814) * Fix * Update * Rename to diffuser and decoder * Define functional model * Merge from upstream/master * Delete old SD3 * Fix copyright * Rename to keras_hub * Address comments * Update * Fix CI * Fix bugs occurred in keras3.1 --------- Co-authored-by: Divyashree Sreepathihalli <divyashreepathihalli@gmail.com> Co-authored-by: Sachin Prasad <sachinprasad@google.com> Co-authored-by: Matt Watson <1389937+mattdangerw@users.noreply.github.com> Co-authored-by: Siva Sravana Kumar Neeli <113718461+sineeli@users.noreply.github.com> Co-authored-by: Usha Rengaraju <34335028+ushareng@users.noreply.github.com> Co-authored-by: ushareng <usha.rengaraju@gmail.com> Co-authored-by: pkgoogle <132095473+pkgoogle@users.noreply.github.com> Co-authored-by: gowthamkpr <47574994+gowthamkpr@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrating and consolidating EfficientNetV1 and V2 to keras-hub.
PR for #1746