Dataset to TFRecord

[cgarciae]

The dataset API is excellent, I am even using it to perform data augmentation before training. However, I notice that when converting my dataset to tfrecord I follow a pattern that might easily be automated as a to_records method, in fact I think this is somewhat similar to using the cache method. Anyway, it would make it very straight forward for dataset users to create tfrecords.

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[mrry]

This is a fine suggestion, and something we have been discussing internally. I'm assigning this issue to @jsimsa, who has expressed interest in developing it.

@cgarciae There are a lot of possible designs for a feature like this. Have you had any thoughts on what your ideal API would look like for this task?

[cgarciae]

@mrry Very glad you are thinking about this!

The signature could be to_records(self, features, filepath, ...) where:

1. It follows the same API of tf.parse_example so you can reuse the same concepts/structures.
2. Should only work on dict-based datasets so you can match the structure in the features dict to the structure on the dataset dict when creating the tf.Example. People who use the tuple-based datasets can just map to a dict beforehand.
3. Create an tf.ImageFeature class that indicates you want to convert the ndarray of an image to bytes encoded in a certain format like jpeg or png.

Btw: I find that tuple-based datasets are harder to maintain when you have many features/columns, the documentation should switch to dict-based approach, or you should even provide a Row class like pyspark does so we can access the columns as fields.

[cgebe]

I guess this functionality would simplify a lot and encourage the usage of TFRecord files. Furthermore, the general documentation on creating/writing to TFRecord files is still sparse.

Was there some progress on this feature request?

[mattiasarro]

@cgarciae Could you post an example of how you convert a Dataset to TFRecords?

[cgarciae]

@mattiasarro here is roughly how I do it right now but not how I would propose:

def create_example(row):
    # create tf.Example(...)

def dataset_generator(ds, sess):
    iterator = ds.make_one_shot_iterator()
    next_row = iterator.get_next()
   
    try:
        while True:
            yield sess.run(next_row)

    except tf.errors.OutOfRangeError:
        pass

def create_records(ds, record_path):
    
    with tf.Session() as sess, tf.python_io.TFRecordWriter(record_path) as writer:

        generator = dataset_generator(ds, sess)
        
        for row in generator:

            example = create_example(row)
            writer.write(example.SerializeToString())

[monstrim]

@cgarciae thanks for that, I'm using it!

I suppose one could pass parse_fn as an argument:

def to_tfrecord(ds, parse_fn, record_path):
    with tf.Session() as sess, tf.python_io.TFRecordWriter(record_path) as writer:
        generator = dataset_generator(ds, sess)
        
        for row in generator:
            example = parse_fn(row)
            writer.write(example.SerializeToString())

[tensorflowbutler]

Nagging Assignee @jsimsa: It has been 179 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.