diff --git a/docs/_book.yaml b/docs/_book.yaml index be10f96a3..ee2daf7a1 100644 --- a/docs/_book.yaml +++ b/docs/_book.yaml @@ -13,25 +13,37 @@ upper_tabs: other: - name: Guide contents: + # Python library usage information + - heading: Using the library - title: Overview path: /hub/overview - - heading: Asset types - - title: SavedModels for TensorFlow 2 - path: /hub/tf2_saved_model - - title: Deprecated: hub.Modules for TensorFlow 1 - path: /hub/tf1_hub_module - - heading: Using the library - - title: Installation + - title: Installation and usage path: /hub/installation - title: Migration to TF2 path: /hub/migration_tf2 - title: Common issues path: /hub/common_issues + - title: Community and support + path: /hub/community + # Asset types available on tfhub.dev + - heading: Asset types + - title: SavedModels for TensorFlow 2 + path: /hub/tf2_saved_model + - title: Deprecated: hub.Modules for TensorFlow 1 + path: /hub/tf1_hub_module + # Contributing to TF-Hub - either models or code + - heading: Contributing + - title: Contribute on GitHub + - path: hub/contribute + - title: Publish models on tfhub.dev + - path: /hub/publish + # Advanced developer topics + - heading: Developer info - title: Build from source path: /hub/build_from_source - - title: Hosting a Module + - title: Hosting your own models path: /hub/hosting - - heading: Tutorials + # Tutorials - title: Text classification - TF2 path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb status: external diff --git a/docs/build_from_source.md b/docs/build_from_source.md index 1095413f7..6e65ab34e 100644 --- a/docs/build_from_source.md +++ b/docs/build_from_source.md @@ -132,7 +132,7 @@ create the necessary python `_pb2.py` files from the `.proto` files. (tensorflow_hub_env)~/hub/$ brew install protobuf ``` -#### On linux +#### On Linux ``` (tensorflow_hub_env)~/hub/$ sudo apt install protobuf-compiler diff --git a/docs/common_issues.md b/docs/common_issues.md index 4519e260e..b3675620e 100644 --- a/docs/common_issues.md +++ b/docs/common_issues.md @@ -1,7 +1,27 @@ # Common issues -If your issue is not listed here, please search the [github issues](https://github.com/tensorflow/hub/issues) before filling a new one. +If your issue is not listed here, please search the +[github issues](https://github.com/tensorflow/hub/issues) before filling a new +one. +## TypeError: 'AutoTrackable' object is not callable + +```python +# BAD: Raises error +embed = hub.load('https://tfhub.dev/google/nnlm-en-dim128/1') +embed(['my text', 'batch']) +``` + +This error frequently arises when loading TF1 hub.Modules with the `hub.load()` +API in TF2. Adding the correct signature should fix this problem. See the +[TF-Hub migration guide for TF2](migration_tf2.md) for more details on moving to +TF2 and the use of TF1 hub.Modules in TF2. + +```python + +embed = hub.load('https://tfhub.dev/google/nnlm-en-dim128/1') +embed.signatures['default'](['my text', 'batch']) +``` ## Cannot download a module @@ -10,21 +30,21 @@ up due to the network stack. Often this is a problem specific to the machine running the code and not an issue with the library. Here is a list of the common ones: -* **"EOF occurred in violation of protocol"** - This issue is likely to be -generated if the installed python version does not support the TLS requirements -of the server hosting the module. Notably, python 2.7.5 is known to fail -resolving modules from tfhub.dev domain. **FIX**: Please update to a newer -python version. +* **"EOF occurred in violation of protocol"** - This issue is likely to be + generated if the installed python version does not support the TLS + requirements of the server hosting the module. Notably, python 2.7.5 is + known to fail resolving modules from tfhub.dev domain. **FIX**: Please + update to a newer python version. -* **"cannot verify tfhub.dev's certificate"** - This issue is likely to be -generated if something on the network is trying to act as the dev gTLD. -Before .dev was used as a gTLD, developers and frameworks would sometimes use -.dev names to help testing code. **FIX:** Identify and reconfigure the software -that intercepts name resolution in the ".dev" domain. +* **"cannot verify tfhub.dev's certificate"** - This issue is likely to be + generated if something on the network is trying to act as the dev gTLD. + Before .dev was used as a gTLD, developers and frameworks would sometimes + use .dev names to help testing code. **FIX:** Identify and reconfigure the + software that intercepts name resolution in the ".dev" domain. If the above errors and fixes do not work, one can try to manually download a -module by simulating the protocol of attaching `?tf-hub-format=compressed` -to the URL to download a tar compressed file that has to be manually decompressed +module by simulating the protocol of attaching `?tf-hub-format=compressed` to +the URL to download a tar compressed file that has to be manually decompressed into a local file. The path to the local file can then be used instead of the URL. Here is a quick example: @@ -49,9 +69,9 @@ Assuming your use-case model is **initialization** and subsequent **requests** (for example Django, Flask, custom HTTP server, etc.), you can set-up the serving as follows: -* In the initialization part: - * Build the graph with a **placeholder** - entry point into the graph. - * Initialize the session. +* In the initialization part: + * Build the graph with a **placeholder** - entry point into the graph. + * Initialize the session. ```python import tensorflow as tf @@ -72,8 +92,8 @@ session = tf.Session(graph=g) session.run(init_op) ``` -* In the request part: - * Use the session to feed data into the graph through the placeholder. +* In the request part: + * Use the session to feed data into the graph through the placeholder. ```python result = session.run(embedded_text, feed_dict={text_input: ["Hello world"]}) diff --git a/docs/community.md b/docs/community.md new file mode 100644 index 000000000..c6168ead2 --- /dev/null +++ b/docs/community.md @@ -0,0 +1,11 @@ +# Community and support + + * Please join us on the [TensorFlow Hub mailing + list](https://groups.google.com/a/tensorflow.org/forum/#!forum/hub) + for announcements, general questions and discussions. + * You can use tag [tensorflow-hub on Stack + Overflow](https://stackoverflow.com/questions/tagged/tensorflow-hub) + for help on how to use TensorFlow Hub. + * The source code is available on [GitHub](https://github.com/tensorflow/hub). + We use [GitHub issues](https://github.com/tensorflow/hub/issues) + for tracking feature requests and bugs. diff --git a/docs/contribute.md b/docs/contribute.md new file mode 100644 index 000000000..fa2db8e7a --- /dev/null +++ b/docs/contribute.md @@ -0,0 +1,4 @@ +# Contribute on GitHub + +Learn more about how to contribute to the `tensorflow/hub` project +[on GitHub](https://github.com/tensorflow/hub/blob/master/CONTRIBUTING.md). diff --git a/docs/hosting.md b/docs/hosting.md index 8cc90eb13..fa63ae5b3 100644 --- a/docs/hosting.md +++ b/docs/hosting.md @@ -1,32 +1,36 @@ -# Hosting a Module +# Hosting your own models -TensorFlow Hub supports HTTP based distribution of modules. In particular -the protocol allows to use the URL identifying the module both as the -documentation of the module and the endpoint to fetch the module. +TensorFlow Hub provides an open repository of trained models at +[thub.dev](https://tfhub.dev). The `tensorflow_hub` library can load models from +this repository and other HTTP based repositories of machine learning models. In +particular the protocol allows to use the URL identifying the model both for the +documentation of the model and the endpoint to fetch the model. +If you are interested in hosting your own repository of models that are loadable +with the `tensorflow_hub` library, your HTTP distribution service should follow +the following protocol. ## Protocol -When a URL such as `https://example.com/module` is used to identify a -module to load or instantiate, the module resolver will attempt to -download a compressed tarball from the URL after appending a query -parameter `?tf-hub-format=compressed`. +When a URL such as `https://example.com/model` is used to identify a model to +load or instantiate, the model resolver will attempt to download a compressed +tarball from the URL after appending a query parameter +`?tf-hub-format=compressed`. -The query param is to be interpreted as a comma separated list of the -module formats that the client is interested in. For now only the -"compressed" format is defined. +The query param is to be interpreted as a comma separated list of the model +formats that the client is interested in. For now only the "compressed" format +is defined. -The **compressed** format indicates that the client expects a `tar.gz` -archive with the module contents. The root of the archive is the root -of the module directory and should contain a SavedModel, as in this -example: +The **compressed** format indicates that the client expects a `tar.gz` archive +with the model contents. The root of the archive is the root of the model +directory and should contain a SavedModel, as in this example: ```shell -# Create a compressed module from a SavedModel directory. -$ tar -cz -f module.tar.gz --owner=0 --group=0 -C /tmp/export-module/ . +# Create a compressed model from a SavedModel directory. +$ tar -cz -f model.tar.gz --owner=0 --group=0 -C /tmp/export-model/ . -# Inspect files inside a compressed module -$ tar -tf module.tar.gz +# Inspect files inside a compressed model +$ tar -tf model.tar.gz ./ ./variables/ ./variables/variables.data-00000-of-00001 @@ -39,6 +43,6 @@ Tarballs for use with the deprecated `hub.Module()` API from TF1 will also contain a `./tfhub_module.pb` file. The `hub.load()` API for TF2 SavedModels ignores such a file. -The `tensorflow_hub` library expects that module URLs are versioned -and that the module content of a given version is immutable, so that it can -be cached indefinitely. +The `tensorflow_hub` library expects that model URLs are versioned and that the +model content of a given version is immutable, so that it can be cached +indefinitely. diff --git a/docs/installation.md b/docs/installation.md index 078a18762..173a85632 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,42 +1,64 @@ -# Installation +# Installation and usage notes -The `tensorflow_hub` library can be installed alongside TensorFlow 1 -and TensorFlow 2. We recommend that new users start with TensorFlow 2 -right away, and current users upgrade to it. +## Installing tensorflow_hub -## Use with TensorFlow 2 +The `tensorflow_hub` library can be installed alongside TensorFlow 1 and +TensorFlow 2. We recommend that new users start with TensorFlow 2 right away, +and current users upgrade to it. + +### Use with TensorFlow 2 Use [pip](https://pip.pypa.io/) to -[install TensorFlow 2](https://www.tensorflow.org/install) as usual. -(See there for extra instructions about GPU support.) -Then install a current version of -[`tensorflow-hub`](https://pypi.org/project/tensorflow-hub/) -next to it (must be 0.5.0 or newer). - +[install TensorFlow 2](https://www.tensorflow.org/install) as usual. (See there +for extra instructions about GPU support.) Then install a current version of +[`tensorflow-hub`](https://pypi.org/project/tensorflow-hub/) next to it (must be +0.5.0 or newer). + ```bash $ pip install "tensorflow>=2.0.0" $ pip install --upgrade tensorflow-hub ``` -The TF1-style API of TensorFlow Hub works with the v1 compatibility mode -of TensorFlow 2. +The TF1-style API of TensorFlow Hub works with the v1 compatibility mode of +TensorFlow 2. -## Legacy use with TensorFlow 1 +### Legacy use with TensorFlow 1 The `tensorflow_hub` library requires TensorFlow version 1.7 or greater. -We strongly recommend to install it with TensorFlow 1.15, which defaults -to TF1-compatible behavior but contains many TF2 features under the hood -to allow some use of TensorFlow Hub's TF2-style APIs. +We strongly recommend to install it with TensorFlow 1.15, which defaults to +TF1-compatible behavior but contains many TF2 features under the hood to allow +some use of TensorFlow Hub's TF2-style APIs. ```bash $ pip install "tensorflow>=1.15,<2.0" $ pip install --upgrade tensorflow-hub ``` -## Use of pre-release versions +### Use of pre-release versions + +The pip packages `tf-nightly` and `tf-hub-nightly` are built automatically from +the source code on github, with no release testing. This lets developers try out +the latest code without [building from source](build_from_source.md). + +## API stability + +Although we hope to prevent breaking changes, this project is still under active +development and is not yet guaranteed to have a stable API or model format. + +## Fairness + +As in all of machine learning, [fairness](http://ml-fairness.com) is an +[important](https://research.googleblog.com/2016/10/equality-of-opportunity-in-machine.html) +consideration. Many pre-trained models are trained on large datasets. When +reusing any model, it’s important to be mindful of what data the model was +trained on (and whether there are any existing biases there), and how these +might impact your use of it. + +## Security -The pip packages `tf-nightly` and `tf-hub-nightly` are built automatically -from the source code on github, with no release testing. This lets -developers try out the latest code without -[building from source](build_from_source.md). +Since they contain arbitrary TensorFlow graphs, models can be thought of as +programs. +[Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md) +describes the security implications of referencing a model from an untrusted +source. diff --git a/docs/overview.md b/docs/overview.md index 92ceff21e..5427649bb 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -1,42 +1,19 @@ # TensorFlow Hub -## Publishing on tfhub.dev - -Are you interested in publishing your modules on tfhub.dev? Express your -interest via our [Publisher -Survey](https://services.google.com/fb/forms/tensorflowhubpublishersurvey/). We -appreciate your valuable feedback, and will be providing more information about -publishing modules in the coming months. For now, please read our documentation -about [Hosting a Module](hosting.md). - -## Fairness - -As in all of machine learning, [fairness](http://ml-fairness.com) is an -[important](https://research.googleblog.com/2016/10/equality-of-opportunity-in-machine.html) -consideration. Modules typically leverage large pretrained datasets. When -reusing such a dataset, it’s important to be mindful of what data it contains -(and whether there are any existing biases there), and how these might impact -your downstream experiments. - - -## Status - -Although we hope to prevent breaking changes, this project is still under active -development and is not yet guaranteed to have a stable API or module format. - - -## Security - -Since they contain arbitrary TensorFlow graphs, modules can be thought of as -programs. [Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md) -describes the security implications of referencing a module from an untrusted -source. - - -## Source-Code & Bug Reports - -The source code is available on [GitHub](https://github.com/tensorflow/hub). -Use [GitHub issues](https://github.com/tensorflow/hub/issues) for feature requests -and bugs. Please see the [TensorFlow Hub mailing -list](https://groups.google.com/a/tensorflow.org/forum/#!forum/hub) for general -questions and discussion. +TensorFlow Hub is a repository and library for reusable machine learning. The +[tfhub.dev](https://tfhub.dev) repository provides many pre-trained models: text +embeddings, image classification models, and more. The `tensorflow_hub` library +lets you download and reuse them in your TensorFlow program with a minimum +amount of code. + +Get started with TensorFlow Hub: + +- [Install TensorFlow Hub](installation.md) +- [TensorFlow Hub in TF2](tf2_saved_model.md) +- [Legacy: TensorFlow Hub in TF1](tf1_hub_module.md) +- Tutorials: + - [Text classification](https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb) + - [Image classification](https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb) + - Additional examples + [on GitHub](https://github.com/tensorflow/hub/blob/master/examples/README.md) +- Find models on [tfhub.dev](https://tfhub.dev) diff --git a/docs/publish.md b/docs/publish.md new file mode 100644 index 000000000..772cdc7f8 --- /dev/null +++ b/docs/publish.md @@ -0,0 +1,7 @@ +# Publishing on tfhub.dev + +Are you interested in publishing your models on tfhub.dev? Express your interest +via our +[Publisher Survey](https://services.google.com/fb/forms/tensorflowhubpublishersurvey/). +We appreciate your valuable feedback and will be providing more information +about publishing modules in the coming months.