TensorFlow-Input-Pipeline

Input Pipeline Examples based on multi-threads and FIFOQueue in TensorFlow, including mini-batching training.

Graphs

Graph for small dataset example code

Graph for big dataset example code

Usage

If your dataset is too large to load once, you can first convert your dataset to TFRecords files, my example code shows how to write data into TFRecords and how to read data from TFRecords correctly. You can run the example code python big_input.py:

usage: small_input.py [-h] [--scale SCALE] [--logdir LOGDIR]
                      [--samples_num SAMPLES_NUM] [--time_length TIME_LENGTH]
                      [--feature_size FEATURE_SIZE] [--num_epochs NUM_EPOCHS]
                      [--batch_size BATCH_SIZE] [--num_classes NUM_CLASSES]

optional arguments:
  -h, --help            show this help message and exit
  --scale SCALE         specify your dataset scale
  --logdir LOGDIR       specify the location to store log or model
  --samples_num SAMPLES_NUM
                        specify your total number of samples
  --time_length TIME_LENGTH
                        specify max time length of sample
  --feature_size FEATURE_SIZE
                        specify feature size of sample
  --num_epochs NUM_EPOCHS
                        specify number of training epochs
  --batch_size BATCH_SIZE
                        specify batch size when training
  --num_classes NUM_CLASSES
                        specify number of output classes

Well, if your dataset is not large enough, you can totally load your dataset once and then use FIFOQueue to improve training. You can also run the example code python small_input.py:

usage: big_input.py [-h] [--scale SCALE] [--logdir LOGDIR]
                    [--samples_num SAMPLES_NUM] [--time_length TIME_LENGTH]
                    [--feature_size FEATURE_SIZE] [--num_epochs NUM_EPOCHS]
                    [--batch_size BATCH_SIZE] [--num_classes NUM_CLASSES]

optional arguments:
  -h, --help            show this help message and exit
  --scale SCALE         specify your dataset scale
  --logdir LOGDIR       specify the location to store log or model
  --samples_num SAMPLES_NUM
                        specify your total number of samples
  --time_length TIME_LENGTH
                        specify max time length of sample
  --feature_size FEATURE_SIZE
                        specify feature size of sample
  --num_epochs NUM_EPOCHS
                        specify number of training epochs
  --batch_size BATCH_SIZE
                        specify batch size when training
  --num_classes NUM_CLASSES
                        specify number of output classes