merge master

README.md

@@ -5,7 +5,7 @@ TensorFlow Datasets provides many public datasets as `tf.data.Datasets`.
 [![Kokoro](https://storage.googleapis.com/tfds-kokoro-public/kokoro-build.svg)](https://storage.googleapis.com/tfds-kokoro-public/kokoro-build.html)
 [![PyPI version](https://badge.fury.io/py/tensorflow-datasets.svg)](https://badge.fury.io/py/tensorflow-datasets)
 
-* [List of datasets](https://github.com/tensorflow/datasets/tree/master/docs/datasets.md)
+* [List of datasets](https://www.tensorflow.org/datasets/catalog/overview)
 * [Try it in Colab](https://colab.research.google.com/github/tensorflow/datasets/blob/master/docs/overview.ipynb)
 * [API docs](https://www.tensorflow.org/datasets/api_docs/python/tfds)
 * Guides

docs/README.md

@@ -1,6 +1,6 @@
 # TensorFlow Datasets
 
-*   [List of datasets](datasets.md)
+*   [List of datasets](https://www.tensorflow.org/datasets/catalog/overview)
 *   [Colab Tutorial](https://github.com/tensorflow/datasets/tree/master/docs/overview.ipynb)
 *   [API Documentation](https://www.tensorflow.org/datasets/api_docs/python/tfds)
 *   [Splits](splits.md)

docs/_book.yaml

@@ -11,7 +11,7 @@ upper_tabs:
   lower_tabs:
     # Subsite tabs
     other:
-    - name: Datasets
+    - name: Catalog
       contents:
       - include: /datasets/catalog/_toc.yaml
     - name: Guide

docs/add_dataset.md

@@ -2,8 +2,8 @@
 
 Follow this guide to add a dataset to TFDS.
 
-See our [list of datasets](datasets.md) to see if the dataset you want isn't
-already added.
+See our [list of datasets](catalog/overview.md) to see if the dataset you want
+isn't already added.
 
 *   [Overview](#overview)
 *   [Writing `my_dataset.py`](#writing-my-datasetpy)

docs/api_docs/python/index.md

@@ -1,5 +1,7 @@
 # All symbols in TensorFlow Datasets
 
+## Primary symbols
+
 *   <a href="./tfds.md"><code>tfds</code></a>
 *   <a href="./tfds/download/GenerateMode.md"><code>tfds.GenerateMode</code></a>
 *   <a href="./tfds/Split.md"><code>tfds.Split</code></a>

docs/api_docs/python/tfds.md

@@ -29,7 +29,7 @@ The main library entrypoints are:
 #### Documentation:
 
 *   These API docs
-*   [Available datasets](https://github.com/tensorflow/datasets/tree/master/docs/datasets.md)
+*   [Available datasets](https://github.com/tensorflow/datasets/tree/master/docs/catalog/overview.md)
 *   [Colab tutorial](https://colab.research.google.com/github/tensorflow/datasets/blob/master/docs/overview.ipynb)
 *   [Add a dataset](https://github.com/tensorflow/datasets/tree/master/docs/add_dataset.md)
 

docs/api_docs/python/tfds/_api_cache.json

@@ -479,6 +479,7 @@
     "tfds.features.Image.get_tensor_info": true, 
     "tfds.features.Image.load_metadata": true, 
     "tfds.features.Image.save_metadata": true, 
+    "tfds.features.Image.set_dtype": true, 
     "tfds.features.Image.set_encoding_format": true, 
     "tfds.features.Image.set_shape": true, 
     "tfds.features.Image.shape": true, 

docs/api_docs/python/tfds/features/Image.md

@@ -10,6 +10,7 @@
 <meta itemprop="property" content="get_tensor_info"/>
 <meta itemprop="property" content="load_metadata"/>
 <meta itemprop="property" content="save_metadata"/>
+<meta itemprop="property" content="set_dtype"/>
 <meta itemprop="property" content="set_encoding_format"/>
 <meta itemprop="property" content="set_shape"/>
 </div>
@@ -54,9 +55,10 @@ Input: The image connector accepts as input:
 <a target="_blank" href="https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/core/features/image_feature.py">View
 source</a>
 
-``` python
+```python
 __init__(
     shape=None,
+    dtype=None,
     encoding_format=None
 )
 ```
@@ -70,6 +72,8 @@ Construct the connector.
     width can be None. For other images: (height, width, channels). height and
     width can be None. See `tf.image.encode_*` for doc on channels parameter.
     Defaults to (None, None, 3).
+*   <b>`dtype`</b>: tf.uint16 or tf.uint8 (default). tf.uint16 can be used only
+    with png encoding_format
 *   <b>`encoding_format`</b>: 'jpeg' or 'png' (default). Format to serialize
     np.ndarray images on disk. If image is loaded from {bmg,gif,jpeg,png} file,
     this parameter is ignored, and file original encoding is used.
@@ -160,6 +164,17 @@ save_metadata(
 
 See base class for details.
 
+<h3 id="set_dtype"><code>set_dtype</code></h3>
+
+<a target="_blank" href="https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/core/features/image_feature.py">View
+source</a>
+
+```python
+set_dtype(dtype)
+```
+
+Update the dtype.
+
 <h3 id="set_encoding_format"><code>set_encoding_format</code></h3>
 
 <a target="_blank" href="https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/core/features/image_feature.py">View

docs/api_docs/python/tfds/load.md

@@ -20,6 +20,7 @@ tfds.load(
     data_dir=None,
     batch_size=None,
     in_memory=None,
+    shuffle_files=None,
     download=True,
     as_supervised=False,
     decoders=None,
@@ -39,6 +40,7 @@ tfds.load(
 
 *   [CycleGAN](https://www.tensorflow.org/beta/tutorials/generative/cyclegan)
 *   [Distributed training with Keras](https://www.tensorflow.org/beta/tutorials/distribute/keras)
+*   [How-to create an Estimator from a Keras model](https://www.tensorflow.org/beta/tutorials/estimators/keras_model_to_estimator)
 *   [Image segmentation](https://www.tensorflow.org/beta/tutorials/images/segmentation)
 *   [Multi-worker Training with Estimator](https://www.tensorflow.org/beta/tutorials/distribute/multi_worker_with_estimator)
 *   [Multi-worker Training with Keras](https://www.tensorflow.org/beta/tutorials/distribute/multi_worker_with_keras)
@@ -102,6 +104,9 @@ of hundreds of GiB to disk. Refer to the `download` argument.
     increases iteration speeds. Note that if `True` and the dataset has unknown
     dimensions, the features will be padded to the maximum size across the
     dataset.
+*   <b>`shuffle_files`</b>: `bool`, whether to shuffle the input files. Defaults
+    to `True` if `split == tfds.Split.TRAIN` and `False` otherwise. From
+    2019-08-20, will always default to False.
 *   <b>`download`</b>: `bool` (optional), whether to call
     <a href="../tfds/core/DatasetBuilder.md#download_and_prepare"><code>tfds.core.DatasetBuilder.download_and_prepare</code></a>
     before calling `tf.DatasetBuilder.as_dataset`. If `False`, data is expected
@@ -131,9 +136,6 @@ of hundreds of GiB to disk. Refer to the `download` argument.
     deduced from data_dir.
 *   <b>`as_dataset_kwargs`</b>: `dict` (optional), keyword arguments passed to
     <a href="../tfds/core/DatasetBuilder.md#as_dataset"><code>tfds.core.DatasetBuilder.as_dataset</code></a>.
-    `split` will be passed through by default. Example: `{'shuffle_files':
-    True}`. Note that shuffle_files is False by default unless `split ==
-    tfds.Split.TRAIN`.
 *   <b>`try_gcs`</b>: `bool`, if True, tfds.load will see if the dataset exists
     on the public GCS bucket before building it locally.
 

docs/api_docs/python/tfds/show_examples.md

@@ -19,7 +19,8 @@ tfds.show_examples(
     ds,
     rows=3,
     cols=3,
-    plot_scale=3.0
+    plot_scale=3.0,
+    image_key=None
 )
 ```
 
@@ -52,6 +53,8 @@ fig = tfds.show_examples(ds_info, ds)
 *   <b>`plot_scale`</b>: `float`, controls the plot size of the images. Keep
     this value around 3 to get a good plot. High and low values may cause the
     labels to get overlapped.
+*   <b>`image_key`</b>: `string`, name of the feature that contains the image.
+    If not set, the system will try to auto-detect it.
 
 #### Returns:
 

docs/api_docs/python/tfds/testing/DatasetBuilderTestCase.md

@@ -1731,7 +1731,7 @@ comparing to zero, or by comparing that the difference between each value
 in the two sequences is more than the given delta.
 
 Note that decimal places (from zero) are usually not the same as significant
-digits (measured from the most signficant digit).
+digits (measured from the most significant digit).
 
 If the two sequences compare equal then they will automatically compare
 almost equal.

docs/api_docs/python/tfds/testing/FeatureExpectationsTestCase.md

@@ -1715,7 +1715,7 @@ comparing to zero, or by comparing that the difference between each value
 in the two sequences is more than the given delta.
 
 Note that decimal places (from zero) are usually not the same as significant
-digits (measured from the most signficant digit).
+digits (measured from the most significant digit).
 
 If the two sequences compare equal then they will automatically compare
 almost equal.

docs/api_docs/python/tfds/testing/SubTestCase.md

@@ -1680,7 +1680,7 @@ or by comparing that the difference between each value in the two sequences is
 more than the given delta.
 
 Note that decimal places (from zero) are usually not the same as significant
-digits (measured from the most signficant digit).
+digits (measured from the most significant digit).
 
 If the two sequences compare equal then they will automatically compare almost
 equal.

docs/api_docs/python/tfds/testing/TestCase.md

@@ -1625,7 +1625,7 @@ comparing to zero, or by comparing that the difference between each value
 in the two sequences is more than the given delta.
 
 Note that decimal places (from zero) are usually not the same as significant
-digits (measured from the most signficant digit).
+digits (measured from the most significant digit).
 
 If the two sequences compare equal then they will automatically compare
 almost equal.

docs/catalog/_toc.yaml

@@ -18,6 +18,10 @@ toc:
     title: binary_alpha_digits
   - path: /datasets/catalog/caltech101
     title: caltech101
+  - path: /datasets/catalog/caltech_birds2010
+    title: caltech_birds2010
+  - path: /datasets/catalog/caltech_birds2011
+    title: caltech_birds2011
   - path: /datasets/catalog/cats_vs_dogs
     title: cats_vs_dogs
   - path: /datasets/catalog/celeb_a
@@ -72,6 +76,8 @@ toc:
     title: kitti
   - path: /datasets/catalog/kmnist
     title: kmnist
+  - path: /datasets/catalog/lfw
+    title: lfw
   - path: /datasets/catalog/lsun
     title: lsun
   - path: /datasets/catalog/mnist
@@ -96,6 +102,8 @@ toc:
     title: resisc45
   - path: /datasets/catalog/rock_paper_scissors
     title: rock_paper_scissors
+  - path: /datasets/catalog/scene_parse150
+    title: scene_parse150
   - path: /datasets/catalog/shapes3d
     title: shapes3d
   - path: /datasets/catalog/smallnorb
@@ -112,6 +120,8 @@ toc:
     title: tf_flowers
   - path: /datasets/catalog/uc_merced
     title: uc_merced
+  - path: /datasets/catalog/visual_domain_decathlon
+    title: visual_domain_decathlon
   - path: /datasets/catalog/voc2007
     title: voc2007
   title: Image

docs/catalog/abstract_reasoning.md

@@ -209,15 +209,13 @@ FeaturesDict({
 ```
 
 ## Statistics
-
 None computed
 
 ## Urls
 
 *   [https://github.com/deepmind/abstract-reasoning-matrices](https://github.com/deepmind/abstract-reasoning-matrices)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `None`
 
 ## Citation

docs/catalog/amazon_us_reviews.md

@@ -1362,7 +1362,6 @@ FeaturesDict({
 ```
 
 ## Statistics
-
 None computed
 
 ## Urls

docs/catalog/bair_robot_pushing_small.md

@@ -22,7 +22,6 @@ including one training set (train) and two test sets of previously seen
 *   Size: `30.06 GiB`
 
 ## Features
-
 ```python
 Sequence({
     'action': Tensor(shape=(4,), dtype=tf.float32),
@@ -45,11 +44,9 @@ TEST  | 256
 *   [https://sites.google.com/view/sna-visual-mpc/](https://sites.google.com/view/sna-visual-mpc/)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `None`
 
 ## Citation
-
 ```
 @misc{1710.05268,
   Author = {Frederik Ebert and Chelsea Finn and Alex X. Lee and Sergey Levine},

docs/catalog/bigearthnet.md

@@ -129,19 +129,16 @@ FeaturesDict({
 ```
 
 ## Statistics
-
 None computed
 
 ## Urls
 
 *   [http://bigearth.net](http://bigearth.net)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `None`
 
 ## Citation
-
 ```
 @article{Sumbul2019BigEarthNetAL,
   title={BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding},

docs/catalog/binarized_mnist.md

@@ -24,7 +24,6 @@ images, so labels are not provided.
 *   Size: `104.68 MiB`
 
 ## Features
-
 ```python
 FeaturesDict({
     'image': Image(shape=(28, 28, 1), dtype=tf.uint8),
@@ -45,11 +44,9 @@ TEST       | 10,000
 *   [http://www.cs.toronto.edu/~larocheh/public/datasets/binarized_mnist/](http://www.cs.toronto.edu/~larocheh/public/datasets/binarized_mnist/)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `None`
 
 ## Citation
-
 ```
 @inproceedings{salakhutdinov2008quantitative,
 title={On the quantitative analysis of deep belief networks},

docs/catalog/binary_alpha_digits.md

@@ -20,7 +20,6 @@ of each class.
 *   Size: `519.83 KiB`
 
 ## Features
-
 ```python
 FeaturesDict({
     'image': Image(shape=(20, 16, 1), dtype=tf.uint8),
@@ -40,11 +39,9 @@ ALL   | 1,404
 *   [https://cs.nyu.edu/~roweis/data/](https://cs.nyu.edu/~roweis/data/)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `(u'image', u'label')`
 
 ## Citation
-
 ```
 
 ```

docs/catalog/caltech101.md

@@ -52,11 +52,9 @@ TRAIN | 3,060
 *   [http://www.vision.caltech.edu/Image_Datasets/Caltech101/](http://www.vision.caltech.edu/Image_Datasets/Caltech101/)
 
 ## Supervised keys (for `as_supervised=True`)
-
 `(u'image', u'label')`
 
 ## Citation
-
 ```
 @article{FeiFei2004LearningGV,
   title={Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories},