tfds.load downloads nsynth with missing parts

[casab]

Short description
I am using tfds.load to download nsynth dataset. But no matter how many times I try, It's only downloaded up until nsynth-train.tfrecord-00975-of-01024. It seems like 976 to 1023 are missing.

Environment information

• Operating System: Windows 10
• Python version: 3.7.3
• tensorflow-datasets/tfds-nightly version: tensorflow-datasets@2.1.0
• tensorflow/tensorflow-gpu/tf-nightly/tf-nightly-gpu version: tensorflow@2.1.0

Reproduction instructions

datasets = tfds.load("nsynth", data_dir="data")

Link to logs

Dl Completed...: 100%|██████████████████████████████████████████████████████████| 989/989 [2:28:10<00:00,  1.89s/ file]

Dl Completed...: 100%|██████████████████████████████████████████████████████████| 989/989 [2:28:10<00:00,  8.99s/ file]

Dataset nsynth downloaded and prepared to data\nsynth\full\2.3.0. Subsequent calls will reuse this data.

Expected behavior
I expect the whole of the dataset to be downloaded.

[Conchylicultor]

This seems to be a bug.
This dataset is pregenerated at: https://pantheon.corp.google.com/storage/browser/tfds-data/datasets/nsynth/full/

It may indicates a bug in our download_gcs_dataset which do not fetch the right number of files.

datasets/tensorflow_datasets/core/utils/gcs_utils.py

Line 81 in 300eb8c

def download_gcs_dataset(

We should check that gcs_files returns the right files and investigate why otherwise (Does GCS has a limit toward the number of file which gets returned).
Don't hesitate if someone want to investigate.

[Conchylicultor]

In the meantime, you could try to copy the datasets manually, using:

gsutil cp tfds-data/datasets/nsynth/full/2.3.0 ~/tensorflow_datasets/nsynth/full/2.3.0

Or you could stream the dataset without downloading with:

ds = tfds.load("nsynth", data_dir="data", try_gcs=True)

[Eshan-Agarwal]

@Conchylicultor What I observed is, it loads all tfrecords but it loads like randomly but loading all of it. I am loading data on my gs bucket, When counting for test tfrecord it loads all 32 examples. Also I am trying for nsynth/gansynth_subset config.

test.tfrecord-00000-of-00032
test.tfrecord-00030-of-00032
test.tfrecord-00004-of-00032
test.tfrecord-00007-of-00032
test.tfrecord-00008-of-00032

@casab Can you count if it loads all examples but loads randomly?

Edit : here is result for test tfrecord printing local paths for test tfrecord

[casab]

@Conchylicultor thanks, I've used the following
gsutil cp -r "gs://tfds-data/datasets/nsynth/full/2.3.0" "~/tensorflow_datasets/nsynth/full/2.3.0"

@Eshan-Agarwal I believe default parameter max_simultaneous_downloads for download_gcs_dataset is 50. That's probably why it downloads 32 of them in random order. Because it's trying to download them in parallel. But the full Nsynth train dataset has 1024 parts. And they are being downloaded randomly by also not exceeding the 50 workers limit. So in the end, parts 976 to 1023 are missing.

[acharles7]

This seems to be a bug.
This dataset is pregenerated at: https://pantheon.corp.google.com/storage/browser/tfds-data/datasets/nsynth/full/

It may indicates a bug in our download_gcs_dataset which do not fetch the right number of files.

datasets/tensorflow_datasets/core/utils/gcs_utils.py

Line 81 in 300eb8c

def download_gcs_dataset(

We should check that gcs_files returns the right files and investigate why otherwise (Does GCS has a limit toward the number of file which gets returned).
Don't hesitate if someone want to investigate.

@Conchylicultor there is a limit on GCS, it returns only the first 1000 files per request if we want more than 1000 we have pass NextMarker (name of last file) and request again until we get all files. This way we can get all files without missing any.

let me know if you want additional information on this. I had implemented this functionality for pg19 dataset to get all files but my problem solved using tf.io.gfile.GFile(file_path, 'r')

[Conchylicultor]

@acharles7 Oh interesting. I didn't knew about the 1000 limitation.

@rsepassi, do you remember why our GCS util is using requests rather than the tf.io.gfile API to list and download files ? Was it performance issue ? Something else ?

[acharles7]

FYI https://cloud.google.com/storage/docs/xml-api/get-bucket-list
The XML API does not return lists longer than 1000 objects.