Dense Prediction API Design, Including Segmentation and Fully Convolutional Networks

[ahundt]

Dense Prediction API Design, Including Segmentation and Fully Convolutional Networks

This issue is to develop an API design for dense prediction tasks such as Segmentation, which includes Fully Convolutional Networks (FCN), and was based on the discussion at #5228 (comment). The goal is to ensure Keras incorporates best practices by default for this sort of problem. Community input, volunteers, and implementations will be very welcome. #6655 is where preprocessing layers can be discussed.

Motivating Tasks and Datasets

• Pascal VOC 2012 Single Label Segmentation
• MSCOCO Multi Label Segmentation
• Unambiguously refer to a particular person or object in image with refcoco
• Reinforcement Learning with OpenAI Gym
• mscoco.org/external has additional examples

Reference Materials

• Fully Convolutional Networks for Semantic Segmentation
• U-Net
• Multi-Scale Context Aggregation by Dilated Convolutions
• ResNet and Resnetv2
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Fully Convolutional DenseNets
• Daniil's Blog (highly detailed, but for tensorflow)
  • FCN for Image Segmentation
  • Image Segmentation with Tensorflow using CNNs and Conditional Random Fields
  • Upsampling and Image Segmentation with Tensorflow and TF-Slim
  • tf-image-segmentation companion repository (tf only)

Feature Requests

These are ideas rather than a finalized proposal so input is welcome!

• Input data: Support one or more Images as input + Supplemental data (ex: image + vector)
• Augmentation of Input Data and Dense Labels
  • Example: Both image and label must be zoomed & translated equally in Pascal VOC
• Input image dimensions should be able to vary
  • Ideally by height, width & number of channels
• Loss function "2D" support, such as single and multi label results for each pixel in an image
• class_weight support for dense labels
  • Example: Single class weight value for each class in an image segmentation task such as in Pascal VOC 2012.
• Sparse to Dense Prediction weight transfer
  • Conversion of ImageNet weights from pre-trained models for segmentation tasks
  • Keras-FCN example
  • Locking of batch normalization layers, often used during transfer process
• Automatic Sparse to Dense Model conversion (advanced)
  • configuration at each downsampling stage
  • remove pooling layers and apply an equivalent atrous dilation in the next convolution layer
  • add an upsampling layer for each downsampling stage
• SegmentationTop Layer?
  • Sigmoid single class predictions
  • Spatial Softmax argmax multi class predictions
  • Multi Label Predictions (sigmoid?)
• "Upsample" Layer?
  • like "Activation" layer, where reasonable upsampling approaches can be defined with a simple string parameter
• Example implementation training & testing on MSCOCO & Pascal VOC 2012 + extended berkeley labels
  • (advanced) pretrain pascal voc on coco then VOC
• COCO pycocotools json format dataset support used by several datasets
  • supports multi-label segmentation, keypoint data, image descriptions, and more
• TFRecord dataset support (probably TensorFlow only, maybe only in tensorflow implementation of keras)
• flow_from_directory & Segmentation Data Generator
  • Keras-FCN,
  • Single class label support
  • Multi class label support
• mean Intesection Over Union (mIOU) utility Keras-FCN
• Image and label masks
• Proper palette handling for png based labels
• sparse label format for multi-label data?
• debugging utilities
  • save predictions to file
• Iterative training of partial networks at varying strides, as described in the FCN paper (advanced, may not be necessary as per Keras-FCN performance)

Existing Keras Utilities with compatible license

• keras-contrib has:
  • DensenetFCN implementation
  • MSCOCO
  • Pascal VOC 2012 + extended berkeley labels
  • a couple of upsampling approaches
  • additional datasets - coco + voc2012 keras-contrib#47 incorporating coco + voc 2012
• Keras-FCN
  • I've been working on this one, current basis for design suggestions
• segmentation_keras
  • includes example using caffe weight conversion utilties
  • fairly clean
• enet-keras
  • includes work towards mscoco support
• https://github.com/azavea/raster-vision/
  • is apache v2 compatible? I think so if keras is in tf now
• https://github.com/JihongJu/keras-fcn

Questions

• Is something as clear as 30 seconds to keras segmentation possible?
• Is anything above missing, redundant, or out of date compared to the state of the art?
• Should the current ImageDataGenerator be extended or is a separate class like Keras-FCN's SegDataGenerator clearer?
• Should there be a guide of some sort?
• What will make for useful training progress and debugging data? (sparse mIOU?, something else?)
• What is needed to handle large datasets quickly and efficiently? (should this be out of scope?)

[fchollet]

Should the current ImageDataGenerator be extended or is a separate class like Keras-FCN's SegDataGenerator clearer?

Depends; if you were to implement it as a subclass, which methods would be reused and which would have to be overridden?

Should there be a guide of some sort?
Is something as clear as 30 seconds to keras segmentation possible?

Sure

What is needed to handle large datasets quickly and efficiently? (should this be out of scope?)

Reading images from disk with ImageDataGenerator using multiprocessing and several processes is already pretty quick and efficient.
The HDF5Matrix can be made more efficient via use of multiprocessing (or at least threading) to avoid IO being a bottleneck.

[Dref360]

Really interested in helping you! Maybe we should have a dedicated slack channel so we could all discuss.

I had a Mean IOU implemented somewhere I'll try to find it!
There is a lot of formats thought on how to specify bounding boxes/ segmentation map. SSD uses Priors shapes, Faster RCNN uses anchors boxes, YOLO v1 is using nothing.
Could get quite crowded.

SSD Keras has some data augmentation for boxes. We could probably uses it.

[ahundt]

@Dref360 the semantic_segmentation slack channel would work. Bounding box design input would be great because I'm not currently using them.

[allanzelener]

I would say that predicting a bounding box is a significantly different task from segmentation, in particular you may need a complicated loss function to handle many boxes. I'm also not sure if best practices are well established enough for this.

For upscaling operations popular choices include:

• Conv2DTranspose. Already implemented in Keras.
• Resize interpolation. See tf.image.resize and discussion on distill.pub. Current UpSampling layer I think does nearest neighbor with integer upsampling factors.
• PixelShuffle/Subpixel Convolution. See this repo for discussion. Now easy in Tensorflow with tf.depth_to_space and some Keras+Theano implementations exist.

Also pix2pix is a popular variant using adversarial training that would be nice to have as an example. There are several Keras implementations out there.

For FCNs I've found base Keras to be pretty useable but one sticking point is that it's not easy to replace a fixed size model or Input layer to one that has None size for all the spatial dimensions, which is all you really need to have a FCN that allows multiple scale inputs. I think the best way to do this now is to create a new instance of the same model except for the Input layer and use get_weights + set_weights. It would be nice if there was a convenient way to just resize the model's input spatial dimensions and have it propagate to all layers, raising an error if it's not possible e.g., if there's a Dense layer.

[PavlosMelissinos]

I'd be interested in contributing as well! However, keep in mind that there are a few subtasks within the segmentation problem and that makes the task harder.

For example, all semantic segmentation networks, such as FCN, segnet, ENet, ICNet etc, do is pixel classification. They cannot detect objects and therefore can't differentiate between distinct instances of the same class in an image.

Other works, such as DeepMask/SharpMask/FastMask, output mask proposals for each object they detect but they do not do classification. This means that in theory they can detect objects that belong in classes they have not seen before.

Finally, Instance Segmentation does both (e.g. Instance-FCN, FCIS, Mask R-CNN). It can tell where a person ends and another begins and also outputs a class label for each instance it detects.

Detection is an inherent part of the pipeline for two of the subtasks, so if we plan to cover all three cases, I don't think we can get away with not discussing it.

[ahundt]

@PavlosMelissinos good points, training on varied tasks like instance recognition and mask proposals should also be considered, what are the best practices for that type of data? How are they typically formatted? Masks are also sometimes useful for segmentation, such as the pascal voc "ambiguous regions".

@Dref360 I thought about the bounding box issue some more and I agree with @allanzelener that the tools will be significantly different for bounding boxes. Unless there is a compelling reason I've missed to keep it here, I think bounding box algorithms should be considered out of scope for this issue and should be handled as a separate github issue.

[ahundt]

For segmentation training it will be important to support loading data data from a directory, and to support the most common dataset formats, which to my knowledge are the Pascal VOC format and the COCO json format. This post goes into loading from a directory in a reasonable way and including support for Pascal VOC.

• I think ImageDataGenerator could simply be updated with couple additional options and parameters
• What would be a good design for loading supplementary data?
• PNG is the obvious choice for multi class single label data
• what support should be provided for multi class multi label data? .npy format seems to work but is too huge for sparse data. Perhaps .mat and .tfrecord formats?

Here is how SegDataGenerator works in Keras-FCN:

seg_aug_generator = SegDataGenerator(
                 featurewise_center=False,
                 samplewise_center=False,
                 featurewise_std_normalization=False,
                 samplewise_std_normalization=False,
                 channelwise_center=False,
                 rotation_range=0.,
                 width_shift_range=0.,
                 height_shift_range=0.,
                 shear_range=0.,
                 zoom_range=0.,
                 zoom_maintain_shape=True,
                 channel_shift_range=0.,
                 fill_mode='constant',
                 cval=0.,
                 label_cval=255,
                 crop_mode='none',
                 crop_size=(0, 0),
                 pad_size=None,
                 horizontal_flip=False,
                 vertical_flip=False,
                 rescale=None,
                 data_format='default')

   generator = seg_aug_generator.flow_from_directory(
                            file_path, data_dir, data_suffix,
                            label_dir, label_suffix, classes,
                            ignore_label=255,
                            target_size=None, color_mode='rgb',
                            class_mode='sparse',
                            batch_size=32, shuffle=True, seed=None,
                            save_to_dir=None, save_prefix='', save_format='jpeg',
                            loss_shape=None)

   model.fit_generator(generator=generator, ...)

# Some internal details for the directory iterator:
    '''
    Users need to ensure that all files exist.
    Label images should be png images where pixel values represents class number.
    find images -name *.jpg > images.txt
    find labels -name *.png > labels.txt
    for a file name 2011_002920.jpg, each row should contain 2011_002920
    file_path: location of train.txt, or val.txt in PASCAL VOC2012 format,
        listing image file path components without extension
    data_dir: location of image files referred to by file in file_path
    label_dir: location of label files
    data_suffix: image file extension, such as `.jpg` or `.png`
    label_suffix: label file suffix, such as `.png`, or `.npy`
    loss_shape: shape to use when applying loss function to the label data
    '''

I think much of this functionality can be added directly to ImageDataGenerator.

file_path becomes file_list

SegDataGenerator.flow_from_directory'sfile_path parameter should be replaced with a file_list parameter and a separate function should be created to ImageDataGenerator that can load a pascal voc formatted .txt file listing filenames without extensions and return a list of them.

Dense class_mode options

class_mode should add new options to specify that it is a dense prediction task. What should these be named, pixel_categorical, pixel_binary, pixel_multilabel etc or perhaps dense instead of pixel? Perhaps it should take a tuple or something else indicating the data dimensionality?

Image Dimensions

Does anyone have design suggestions for dealing with the dimension issue detailed by @allanzelener? The SegDataGenerator design is to simply pad images with mask pixels to the maximum expected image size. This seems to work okay, but can probably have significant computational cost.

new augmentation options

I think most of the new augmentation options in SegDataGenerator also look good and can simply be added directly.

Supplementary Data

Supplementary Data is also likely necessary (definitely in my case), I think it may be wise to allow a second list of input files to be supplied in a different format, which can be simple vectors or images stored in a .mat or .npy, or some other format. However, perhaps this should be a separate class? If so, how would consistency of indexes be ensured? Can two different generators be chained together in a manner analogous to zip() for lists?

loss_shape

loss_shape is a workaround, because the output dimensions will vary based on the model, and we will want the loss function to operate on the output data as it is. Can it be avoided?

File Formats

Common Formats

In the current ImageDataGenerator:

Any PNG, JPG or BMP images inside each of the subdirectories directory tree will be included in the generator.

I think the addition of data_dir, data_suffix, label_dir, label_suffix is a good decision that does not need to conflict with this, they can simply default to None which retains the current behavior.

Arbitrary Formats

The API could easily support arbitrary file formats with a function, object that opens the files in the directory and returns an appropriate numpy array. Should this exist? Which parameter should accept these? Perhaps instead of *_suffix the parameter could be *_format, which can take these classes and/or functions? Design suggestions welcome