Skip to content

Commit

Permalink
Reorganize docs on community, hosting, installation for TF2
Browse files Browse the repository at this point in the history
PiperOrigin-RevId: 273956378
  • Loading branch information
TensorFlow Hub Authors authored and andresusanopinto committed Oct 14, 2019
1 parent 0b0a4e9 commit 662e23b
Show file tree
Hide file tree
Showing 9 changed files with 169 additions and 112 deletions.
30 changes: 21 additions & 9 deletions docs/_book.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,25 +13,37 @@ upper_tabs:
other:
- name: Guide
contents:
# Python library usage information
- heading: Using the library
- title: Overview
path: /hub/overview
- heading: Asset types
- title: SavedModels for TensorFlow 2
path: /hub/tf2_saved_model
- title: Deprecated: hub.Modules for TensorFlow 1
path: /hub/tf1_hub_module
- heading: Using the library
- title: Installation
- title: Installation and usage
path: /hub/installation
- title: Migration to TF2
path: /hub/migration_tf2
- title: Common issues
path: /hub/common_issues
- title: Community and support
path: /hub/community
# Asset types available on tfhub.dev
- heading: Asset types
- title: SavedModels for TensorFlow 2
path: /hub/tf2_saved_model
- title: Deprecated: hub.Modules for TensorFlow 1
path: /hub/tf1_hub_module
# Contributing to TF-Hub - either models or code
- heading: Contributing
- title: Contribute on GitHub
- path: hub/contribute
- title: Publish models on tfhub.dev
- path: /hub/publish
# Advanced developer topics
- heading: Developer info
- title: Build from source
path: /hub/build_from_source
- title: Hosting a Module
- title: Hosting your own models
path: /hub/hosting
- heading: Tutorials
# Tutorials
- title: Text classification - TF2
path: https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb
status: external
Expand Down
2 changes: 1 addition & 1 deletion docs/build_from_source.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ create the necessary python `_pb2.py` files from the `.proto` files.
(tensorflow_hub_env)~/hub/$ brew install protobuf
```

#### On linux
#### On Linux

```
(tensorflow_hub_env)~/hub/$ sudo apt install protobuf-compiler
Expand Down
56 changes: 38 additions & 18 deletions docs/common_issues.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,27 @@
# Common issues

If your issue is not listed here, please search the [github issues](https://github.com/tensorflow/hub/issues) before filling a new one.
If your issue is not listed here, please search the
[github issues](https://github.com/tensorflow/hub/issues) before filling a new
one.

## TypeError: 'AutoTrackable' object is not callable

```python
# BAD: Raises error
embed = hub.load('https://tfhub.dev/google/nnlm-en-dim128/1')
embed(['my text', 'batch'])
```

This error frequently arises when loading TF1 hub.Modules with the `hub.load()`
API in TF2. Adding the correct signature should fix this problem. See the
[TF-Hub migration guide for TF2](migration_tf2.md) for more details on moving to
TF2 and the use of TF1 hub.Modules in TF2.

```python

embed = hub.load('https://tfhub.dev/google/nnlm-en-dim128/1')
embed.signatures['default'](['my text', 'batch'])
```

## Cannot download a module

Expand All @@ -10,21 +30,21 @@ up due to the network stack. Often this is a problem specific to the machine
running the code and not an issue with the library. Here is a list of the common
ones:

* **"EOF occurred in violation of protocol"** - This issue is likely to be
generated if the installed python version does not support the TLS requirements
of the server hosting the module. Notably, python 2.7.5 is known to fail
resolving modules from tfhub.dev domain. **FIX**: Please update to a newer
python version.
* **"EOF occurred in violation of protocol"** - This issue is likely to be
generated if the installed python version does not support the TLS
requirements of the server hosting the module. Notably, python 2.7.5 is
known to fail resolving modules from tfhub.dev domain. **FIX**: Please
update to a newer python version.

* **"cannot verify tfhub.dev's certificate"** - This issue is likely to be
generated if something on the network is trying to act as the dev gTLD.
Before .dev was used as a gTLD, developers and frameworks would sometimes use
.dev names to help testing code. **FIX:** Identify and reconfigure the software
that intercepts name resolution in the ".dev" domain.
* **"cannot verify tfhub.dev's certificate"** - This issue is likely to be
generated if something on the network is trying to act as the dev gTLD.
Before .dev was used as a gTLD, developers and frameworks would sometimes
use .dev names to help testing code. **FIX:** Identify and reconfigure the
software that intercepts name resolution in the ".dev" domain.

If the above errors and fixes do not work, one can try to manually download a
module by simulating the protocol of attaching `?tf-hub-format=compressed`
to the URL to download a tar compressed file that has to be manually decompressed
module by simulating the protocol of attaching `?tf-hub-format=compressed` to
the URL to download a tar compressed file that has to be manually decompressed
into a local file. The path to the local file can then be used instead of the
URL. Here is a quick example:

Expand All @@ -49,9 +69,9 @@ Assuming your use-case model is **initialization** and subsequent **requests**
(for example Django, Flask, custom HTTP server, etc.), you can set-up the
serving as follows:

* In the initialization part:
* Build the graph with a **placeholder** - entry point into the graph.
* Initialize the session.
* In the initialization part:
* Build the graph with a **placeholder** - entry point into the graph.
* Initialize the session.

```python
import tensorflow as tf
Expand All @@ -72,8 +92,8 @@ session = tf.Session(graph=g)
session.run(init_op)
```

* In the request part:
* Use the session to feed data into the graph through the placeholder.
* In the request part:
* Use the session to feed data into the graph through the placeholder.

```python
result = session.run(embedded_text, feed_dict={text_input: ["Hello world"]})
Expand Down
11 changes: 11 additions & 0 deletions docs/community.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Community and support

* Please join us on the [TensorFlow Hub mailing
list](https://groups.google.com/a/tensorflow.org/forum/#!forum/hub)
for announcements, general questions and discussions.
* You can use tag [tensorflow-hub on Stack
Overflow](https://stackoverflow.com/questions/tagged/tensorflow-hub)
for help on how to use TensorFlow Hub.
* The source code is available on [GitHub](https://github.com/tensorflow/hub).
We use [GitHub issues](https://github.com/tensorflow/hub/issues)
for tracking feature requests and bugs.
4 changes: 4 additions & 0 deletions docs/contribute.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Contribute on GitHub

Learn more about how to contribute to the `tensorflow/hub` project
[on GitHub](https://github.com/tensorflow/hub/blob/master/CONTRIBUTING.md).
48 changes: 26 additions & 22 deletions docs/hosting.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,36 @@
# Hosting a Module
# Hosting your own models

TensorFlow Hub supports HTTP based distribution of modules. In particular
the protocol allows to use the URL identifying the module both as the
documentation of the module and the endpoint to fetch the module.
TensorFlow Hub provides an open repository of trained models at
[thub.dev](https://tfhub.dev). The `tensorflow_hub` library can load models from
this repository and other HTTP based repositories of machine learning models. In
particular the protocol allows to use the URL identifying the model both for the
documentation of the model and the endpoint to fetch the model.

If you are interested in hosting your own repository of models that are loadable
with the `tensorflow_hub` library, your HTTP distribution service should follow
the following protocol.

## Protocol

When a URL such as `https://example.com/module` is used to identify a
module to load or instantiate, the module resolver will attempt to
download a compressed tarball from the URL after appending a query
parameter `?tf-hub-format=compressed`.
When a URL such as `https://example.com/model` is used to identify a model to
load or instantiate, the model resolver will attempt to download a compressed
tarball from the URL after appending a query parameter
`?tf-hub-format=compressed`.

The query param is to be interpreted as a comma separated list of the
module formats that the client is interested in. For now only the
"compressed" format is defined.
The query param is to be interpreted as a comma separated list of the model
formats that the client is interested in. For now only the "compressed" format
is defined.

The **compressed** format indicates that the client expects a `tar.gz`
archive with the module contents. The root of the archive is the root
of the module directory and should contain a SavedModel, as in this
example:
The **compressed** format indicates that the client expects a `tar.gz` archive
with the model contents. The root of the archive is the root of the model
directory and should contain a SavedModel, as in this example:

```shell
# Create a compressed module from a SavedModel directory.
$ tar -cz -f module.tar.gz --owner=0 --group=0 -C /tmp/export-module/ .
# Create a compressed model from a SavedModel directory.
$ tar -cz -f model.tar.gz --owner=0 --group=0 -C /tmp/export-model/ .

# Inspect files inside a compressed module
$ tar -tf module.tar.gz
# Inspect files inside a compressed model
$ tar -tf model.tar.gz
./
./variables/
./variables/variables.data-00000-of-00001
Expand All @@ -39,6 +43,6 @@ Tarballs for use with the deprecated `hub.Module()` API from TF1 will also
contain a `./tfhub_module.pb` file. The `hub.load()` API for TF2 SavedModels
ignores such a file.

The `tensorflow_hub` library expects that module URLs are versioned
and that the module content of a given version is immutable, so that it can
be cached indefinitely.
The `tensorflow_hub` library expects that model URLs are versioned and that the
model content of a given version is immutable, so that it can be cached
indefinitely.
66 changes: 44 additions & 22 deletions docs/installation.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,64 @@
# Installation
# Installation and usage notes

The `tensorflow_hub` library can be installed alongside TensorFlow 1
and TensorFlow 2. We recommend that new users start with TensorFlow 2
right away, and current users upgrade to it.
## Installing tensorflow_hub

## Use with TensorFlow 2
The `tensorflow_hub` library can be installed alongside TensorFlow 1 and
TensorFlow 2. We recommend that new users start with TensorFlow 2 right away,
and current users upgrade to it.

### Use with TensorFlow 2

Use [pip](https://pip.pypa.io/) to
[install TensorFlow 2](https://www.tensorflow.org/install) as usual.
(See there for extra instructions about GPU support.)
Then install a current version of
[`tensorflow-hub`](https://pypi.org/project/tensorflow-hub/)
next to it (must be 0.5.0 or newer).

[install TensorFlow 2](https://www.tensorflow.org/install) as usual. (See there
for extra instructions about GPU support.) Then install a current version of
[`tensorflow-hub`](https://pypi.org/project/tensorflow-hub/) next to it (must be
0.5.0 or newer).

```bash
$ pip install "tensorflow>=2.0.0"
$ pip install --upgrade tensorflow-hub
```

The TF1-style API of TensorFlow Hub works with the v1 compatibility mode
of TensorFlow 2.
The TF1-style API of TensorFlow Hub works with the v1 compatibility mode of
TensorFlow 2.

## Legacy use with TensorFlow 1
### Legacy use with TensorFlow 1

The `tensorflow_hub` library requires TensorFlow version 1.7 or greater.

We strongly recommend to install it with TensorFlow 1.15, which defaults
to TF1-compatible behavior but contains many TF2 features under the hood
to allow some use of TensorFlow Hub's TF2-style APIs.
We strongly recommend to install it with TensorFlow 1.15, which defaults to
TF1-compatible behavior but contains many TF2 features under the hood to allow
some use of TensorFlow Hub's TF2-style APIs.

```bash
$ pip install "tensorflow>=1.15,<2.0"
$ pip install --upgrade tensorflow-hub
```

## Use of pre-release versions
### Use of pre-release versions

The pip packages `tf-nightly` and `tf-hub-nightly` are built automatically from
the source code on github, with no release testing. This lets developers try out
the latest code without [building from source](build_from_source.md).

## API stability

Although we hope to prevent breaking changes, this project is still under active
development and is not yet guaranteed to have a stable API or model format.

## Fairness

As in all of machine learning, [fairness](http://ml-fairness.com) is an
[important](https://research.googleblog.com/2016/10/equality-of-opportunity-in-machine.html)
consideration. Many pre-trained models are trained on large datasets. When
reusing any model, it’s important to be mindful of what data the model was
trained on (and whether there are any existing biases there), and how these
might impact your use of it.

## Security

The pip packages `tf-nightly` and `tf-hub-nightly` are built automatically
from the source code on github, with no release testing. This lets
developers try out the latest code without
[building from source](build_from_source.md).
Since they contain arbitrary TensorFlow graphs, models can be thought of as
programs.
[Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md)
describes the security implications of referencing a model from an untrusted
source.
57 changes: 17 additions & 40 deletions docs/overview.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,19 @@
# TensorFlow Hub

## Publishing on tfhub.dev

Are you interested in publishing your modules on tfhub.dev? Express your
interest via our [Publisher
Survey](https://services.google.com/fb/forms/tensorflowhubpublishersurvey/). We
appreciate your valuable feedback, and will be providing more information about
publishing modules in the coming months. For now, please read our documentation
about [Hosting a Module](hosting.md).

## Fairness

As in all of machine learning, [fairness](http://ml-fairness.com) is an
[important](https://research.googleblog.com/2016/10/equality-of-opportunity-in-machine.html)
consideration. Modules typically leverage large pretrained datasets. When
reusing such a dataset, it’s important to be mindful of what data it contains
(and whether there are any existing biases there), and how these might impact
your downstream experiments.


## Status

Although we hope to prevent breaking changes, this project is still under active
development and is not yet guaranteed to have a stable API or module format.


## Security

Since they contain arbitrary TensorFlow graphs, modules can be thought of as
programs. [Using TensorFlow Securely](https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md)
describes the security implications of referencing a module from an untrusted
source.


## Source-Code & Bug Reports

The source code is available on [GitHub](https://github.com/tensorflow/hub).
Use [GitHub issues](https://github.com/tensorflow/hub/issues) for feature requests
and bugs. Please see the [TensorFlow Hub mailing
list](https://groups.google.com/a/tensorflow.org/forum/#!forum/hub) for general
questions and discussion.
TensorFlow Hub is a repository and library for reusable machine learning. The
[tfhub.dev](https://tfhub.dev) repository provides many pre-trained models: text
embeddings, image classification models, and more. The `tensorflow_hub` library
lets you download and reuse them in your TensorFlow program with a minimum
amount of code.

Get started with TensorFlow Hub:

- [Install TensorFlow Hub](installation.md)
- [TensorFlow Hub in TF2](tf2_saved_model.md)
- [Legacy: TensorFlow Hub in TF1](tf1_hub_module.md)
- Tutorials:
- [Text classification](https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb)
- [Image classification](https://github.com/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb)
- Additional examples
[on GitHub](https://github.com/tensorflow/hub/blob/master/examples/README.md)
- Find models on [tfhub.dev](https://tfhub.dev)
7 changes: 7 additions & 0 deletions docs/publish.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Publishing on tfhub.dev

Are you interested in publishing your models on tfhub.dev? Express your interest
via our
[Publisher Survey](https://services.google.com/fb/forms/tensorflowhubpublishersurvey/).
We appreciate your valuable feedback and will be providing more information
about publishing modules in the coming months.

0 comments on commit 662e23b

Please sign in to comment.