HDF5 output

[normanrz]

Store frames/flow data in HDF5 format

[normanrz]

Ask Christian and Hannes

[tmbo]

BVLC/caffe#1698 might be useful. It seems we can also use LMDB for label storage. We would then create a separate lmdb that only contains the labels

[fahimeh62]

@normanrz Sorry, I am new in Caffe. How do you store your data in HDF5 format. I have flow matrix that has 2 components (I generated it by Matlab) and I am going to import it as a HDF5 file to Caffe.

[tmbo]

Please use the caffe user mailing list for questions about caffe https://groups.google.com/forum/#!forum/caffe-users.

Nevertheless, here is what we did:

• stack the images of the flow to get a 4 dimensional array: dims [n, flow-depth, height, width] where n is the number of flow stacks and flow-depth is the number flow images you calculated for the current image
• store the array in hdf5 (https://github.com/BVLC/caffe/blob/master/matlab/hdf5creation/store2hdf5.m)

[fahimeh62]

@tmbo Thanks for your answer. Actually, I did what you suggested and I successfully stacked my images in a 4D Array, but when I fed it into Caffe using HDFDATA layer, the accuracy was very low compared with feeding images separately using ImageData layer. Did you load your array using HDF5DATA layer as well?
Thanks for your help.

[hotzenklotz]

@fahimeh62 Please use the Caffe Mailing list for better support.

You can check our code to save data as HDF5 here: https://github.com/normanrz/face-vid/blob/master/frameIO.py#L98 Make sure that your data is in the correct order for Caffe: `Frames x Layers x X x Y as one big nparray``

You can check our network to see how we load the HDF5 layer. There is nothing special here: https://github.com/normanrz/face-vid/blob/master/face-vid-nets/one-vs-all/train_val.prototxt#L2
Make sure your data is correctly saved in the HDF5 DB. Use the HDF5 Viewer to check. It supports a live image preview of the HDF5 data: https://www.hdfgroup.org/products/java/release/download.html

[fahimeh62]

@hotzenklotz I rechecked my data. when I view my data in HDF5 viewer, the dimension of data is: 7996_2_224_244 and the dimension of label is 7996_1 in my case. I fed this data into Caffe using a txt file including the path of my (.h5) file. Are the dimensions right?
I can view my data, it contains images and labels. It seems fine.
I used the same process as you in my network about defining hdf5data layers.
I will be thankful if you can tell me what was wrong.
Cheers.