Added the unpooling layer, that does unpooling operation

[nyf-nyf]

I've added the unpooling layer. It's described in articles of using Deconvolutional Neural Networks.

http://www.matthewzeiler.com/pubs/iccv2011/iccv2011.pdf

But i'm not expert in c++, would appreciate any fixes.
The version for cuda is missing, but works fine for me with cpu.

It helps to debug a convolutional network from bottom to top using unpooling, relu and deconvolution.

[Trekky12]

Hey @nyf-nyf,

unfortunately there is no example for deconv networks. Can you explain the debugging with unpooling, relu and deconvolution?

Thank!

[nyf-nyf]

I can show you the example of the net, that I'm working with. So the mechanism is simple. Each pooling layer outputs the value and the mask. I use mask in unpooling layers to recover the image.
The example below shows how I recover image from the "conv5" layer. But you can pefrorm this operation on other convolutional layers, for example on 3rd - just change the "bottom" value of "conv3t" layer to "conv3".

name: "CaffeNet"
layer {
  name: "memory"
  type: "MemoryData"
  top: "data"
  top: "label"
  memory_data_param {
    batch_size: 1
    channels: 3
    height: 227
    width: 227
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  top: "pool1_mask"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm1"
  type: "LRN"
  bottom: "pool1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "norm1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  top: "pool2_mask"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm2"
  type: "LRN"
  bottom: "pool2"
  top: "norm2"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "norm2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu4"
  type: "ReLU"
  bottom: "conv4"
  top: "conv4"
}
layer {
  name: "conv5"
  type: "Convolution"
  bottom: "conv4"
  top: "conv5"
  param {
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu5"
  type: "ReLU"
  bottom: "conv5"
  top: "conv5"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5"
  top: "pool5"
  top: "pool5_mask"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 10
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "fc8"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}
layer {
  name: "prob"
  type: "Softmax"
  bottom: "fc8"
  top: "prob"
}
# DECONVOLUTION PART
layer {
  name: "unpool5"
  type: "UnPooling"
  bottom: "pool5"
  bottom: "pool5_mask"
  top: "unpool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "relu5t"
  type: "ReLU"
  bottom: "unpool5"
  top: "unpool5"
}
layer {
  name: "conv5t"
  type: "Deconvolution"
  bottom: "unpool5"
  top: "conv5t"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layer {
  name: "relu4t"
  type: "ReLU"
  bottom: "conv5t"
  top: "conv5t"
}
layer {
  name: "conv4t"
  type: "Deconvolution"
  bottom: "conv5t"
  top: "conv4t"
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layer {
  name: "relu3t"
  type: "ReLU"
  bottom: "conv4t"
  top: "conv4t"
}
layer {
  name: "conv3t"
  type: "Deconvolution"
  bottom: "conv4t"
  top: "conv3t"
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
  }
}
layer {
  name: "unpool2"
  type: "UnPooling"
  bottom: "conv3t"
  bottom: "pool2_mask"
  top: "unpool2"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "relu2t"
  type: "ReLU"
  bottom: "unpool2"
  top: "unpool2"
}
layer {
  name: "conv2t"
  type: "Deconvolution"
  bottom: "unpool2"
  top: "conv2t"
  convolution_param {
    num_output: 96
    pad: 2
    kernel_size: 5
    group: 2
  }
}
layer {
  name: "unpool1"
  type: "UnPooling"
  bottom: "conv2t"
  bottom: "pool1_mask"
  top: "unpool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "relu1t"
  type: "ReLU"
  bottom: "unpool1"
  top: "unpool1"
}
layer {
  name: "conv1t"
  type: "Deconvolution"
  bottom: "unpool1"
  top: "conv1t"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 3
    kernel_size: 11
    stride: 4
  }
}

I'm using python and opencv. So I simply make all feature maps on "pool5" layer to zero except one. And I can see the activations of this one only.

[Trekky12]

Sorry for the late reply. Thanks for the example net, but I don't really get how you make all feature maps on "pool5" to zero except one. Do you forward propagate an image and then do the net surgery? Can you please explain the python example?

[nyf-nyf]

Just like that

I make a copy of layer params
filters = solver.net.params["conv5"][0].data.copy()
and then
solver.net.params["conv5"][0].data[...] = 0
solver.net.params["conv5"][0].data[YOUR_FILTER_NUMBER] = filters[YOUR_FILTER_NUMBER]

[seanbell]

Why is the backwards pass just a return statement? The backwards pass should be simple and similar to the forwards pass.

Also, it doesn't make sense to have a pooling type. There is no "MAX" pooling -- it simply is taking in the indices from bottom[1].

[ahaque]

@nyf-nyf If the conv5 and conv5t kernel size is 3, how are you generating the image in the conv1t opencv window you posted?

[Trekky12]

@nyf-nyf thank you for your help.

I tried to use your example and fine-tuned my net with the modified prototxt with the Deconvolution and UnPooling Layers. After obtaining the weights file I tried to load in with the help of the new Matlab interface.
Unfortunately the returned image is full of zeros. Do you know why?

I did the following in Matlab:

model_file = 'deconvnet_deploy.prototxt';
weights_file = 'caffenet_finetune_4_iter_1000.caffemodel';
im_data = caffe.io.load_image('testimage.bmp');
im_data = imresize(im_data, [256 256], 'bilinear','AntiAliasing',false); 
im_center_crop = im_data(15:241, 15:241, :); 

filters_orig = net.params('conv5', 1).get_data();
filters_mod = zeros(net.params('conv5', 1).shape);
filters_mod(:,:,:,1) = filters_orig(:,:,:,1);
net.params('conv5', 1).set_data(filters_mod);

res = net.forward({im_center_crop});
deconvimage = res{1};

I've removed the MemoryData and use the following instead:

input: "data"
input_dim: 10
input_dim: 3
input_dim: 227
input_dim: 227

I've also removed the accuracy and SoftmaxWithLoss Layer for the prediction.
Is there anything more neccessary?

After some testing I recognized, that the weights and data of the conv*t layers are zero. How can this be avoided?

Thanks,
Trekky

[nyf-nyf]

@Trekky12 according to the documentation mensioned at the top, the weights of Deconvolutional Layers are made of transposed weights of the same Conv layers.

In python i manually apply them before starting the net training or testing like that

solver.net.params["conv1t"][0].data[...] = solver.net.params["conv1"][0].data.transpose(0,1,3,2)
solver.net.params["conv2t"][0].data[...] = solver.net.params["conv2"][0].data.transpose(0,1,3,2)
solver.net.params["conv3t"][0].data[...] = solver.net.params["conv3"][0].data.transpose(0,1,3,2)
solver.net.params["conv4t"][0].data[...] = solver.net.params["conv4"][0].data.transpose(0,1,3,2)
solver.net.params["conv5t"][0].data[...] = solver.net.params["conv5"][0].data.transpose(0,1,3,2)

[nyf-nyf]

@ahaque
Try to use that class, i've made it to show the contents of blobs in opencv

class bb(object):
    padding = 5
    max_width = 200 # Max width of an opencv window
    max_height = 100 # Max height of an opencv window 
    display_all = True
    def __init__(self, net, blobname):
        self.net = net
        self.blobname = blobname
        self.cols = self.net.blobs[self.blobname].channels
        self.rows = self.net.blobs[self.blobname].num
        if self.display_all is False:
            if self.net.blobs[self.blobname].channels*self.net.blobs[self.blobname].width > self.max_width:
                self.cols = self.net.blobs[self.blobname].channels - ((self.net.blobs[self.blobname].channels*self.net.blobs[self.blobname].width - self.max_width) / self.net.blobs[self.blobname].width)
            if self.net.blobs[self.blobname].num*self.net.blobs[self.blobname].height > self.max_height:
                self.rows -= ((self.net.blobs[self.blobname].num*self.net.blobs[self.blobname].height - self.max_height) / self.net.blobs[self.blobname].height)
        self.width = self.cols*self.net.blobs[self.blobname].width+((self.cols-1)*self.padding)
        self.height = self.rows*self.net.blobs[self.blobname].height+((self.rows-1)*self.padding)
        self.data = np.zeros((self.height, self.width), self.net.blobs[self.blobname].data.dtype)

    def get_3color(self):
        self.data = np.zeros((self.height, self.net.blobs[self.blobname].width, 3), self.net.blobs[self.blobname].data.dtype)
        for row in range(0, self.rows):
            d = self.net.blobs[self.blobname].data[row].copy()
            d -= d.min()
            d /= d.max()
            d *= 255
            d = np.rollaxis(d, 0, 3)
            self.data[(row*self.net.blobs[self.blobname].height)+(row*self.padding):(row*self.net.blobs[self.blobname].height)+(row*self.padding)+self.net.blobs[self.blobname].height] = d
        return self.data

    def get_image(self):
        self.data = np.zeros((self.height, self.width), self.net.blobs[self.blobname].data.dtype)
        for row in range(0, self.rows):
            for col in range(0, self.cols):
                d = self.net.blobs[self.blobname].data[row][col].copy()
                d -= d.min()
                d /= d.max()
                d *= 255
                self.data[(row*self.net.blobs[self.blobname].height)+(row*self.padding):(row*self.net.blobs[self.blobname].height)+(row*self.padding)+self.net.blobs[self.blobname].height,(col*self.net.blobs[self.blobname].width)+(col*self.padding):(col*self.net.blobs[self.blobname].width)+(col*self.padding)+self.net.blobs[self.blobname].width] = d
        return self.data

It is initialized with 2 params. The net and the name of the blob.
For example blob_conv1t = bb(solver.net, "conv1t")

After net forwarding you can show the layers blob like that:
if blob has 3 channels:
cv2.imshow("conv1t", blob_conv1t.get_3color().astype(np.uint8))
if blob has more than 3 channels (it would be one image for channel in a row)
cv2.imshow("conv2t", blob_conv2t.get_image().astype(np.uint8))

[nyf-nyf]

@seanbell i've made the backwards method just to return, because the Deconvolutional layers, as I understood, are not used in training. Just to "debug" the net.
If I'm wrong, it can be modified in future commits. I just made this pull req to help others to dig deeper in the nets :-)

[Trekky12]

@nyf-nyf thank you for this hint but neither in Matlab nor in python I can create an image. The image remains zeros.

caffe.set_mode_cpu()
net = caffe.Net('deconvnet_deploy.prototxt', 'caffenet_finetune_4_iter_2000.caffemodel', caffe.TEST)

transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
transformer.set_transpose('data', (2,0,1))
transformer.set_raw_scale('data', 255) 
transformer.set_channel_swap('data', (2,1,0)) 
image = transformer.preprocess('data', caffe.io.load_image('testimage.bmp'))
net.blobs['data'].data[...] = image

net.params["conv1t"][0].data[...] = net.params["conv1"][0].data.transpose(0,1,3,2)
net.params["conv2t"][0].data[...] = net.params["conv2"][0].data.transpose(0,1,3,2)
net.params["conv3t"][0].data[...] = net.params["conv3"][0].data.transpose(0,1,3,2)
net.params["conv4t"][0].data[...] = net.params["conv4"][0].data.transpose(0,1,3,2)
net.params["conv5t"][0].data[...] = net.params["conv5"][0].data.transpose(0,1,3,2)

filters = net.params["conv5"][0].data.copy()
net.params["conv5"][0].data[...] = 0
net.params["conv5"][0].data[0] = filters[0]

out = net.forward()

Do you know the problem?
Thank you very much for your help!

Trekky

Edit: Apparently there is indeed an image generated. I tested it with my own net and the cat.jpg testimage and the bvlc_reference_caffenet.caffemodel but I don't think the output is correct. Unfortunately the filter number doesn't matter because the same image is always generated.

[ahaque]

@nyf-nyf Thank you!

[nyf-nyf]

@Trekky12 after forwarding can u please check the values of the blob. Max and min are 0?

net.blobs["conv1t"].data.max()
net.blobs["conv1t"].data.min()

If they are 0, check please the conv5 blob

net.blobs["conv5"].data.max() and min()

Are these values are 0 and 0?

[Trekky12]

@nyf-nyf Thank you for your quick response.

The max and min are not 0:

net.blobs["conv1t"].data.max(): 2.03624
net.blobs["conv1t"].data.min(): -2.28395
net.blobs["conv5"].data.max(): 12.7655
net.blobs["conv5"].data.min(): 0.0

[nyf-nyf]

@Trekky12 how are you viewing the image of the blob?
For example the opencv method "cv2.imshow(name, image)" gets the values between 0 and 255 in bgr format.

May be the problem is in displaying the image?

Your conv1t blob layer has values from -2 to 2. You need to convert them from 0 to 255.
Try to use my class "bb" to display the image.

BTW do not forget to convert the type of an array to np.uint8 like that cv2.imshow("window name", image.astype(np.uint8)). Because opencv method cv2.imshow doesn't work with np.float32 values.

[Trekky12]

@nyf-nyf I think this was the problem. I did that and the image is indeed a little bit different and seems more accurate. Do you think this is correct for the cat.jpg testimage? Why is there the gray border around the structure?

[nyf-nyf]

@Trekky12 as i saw in your previous messages, you enabled only the first filter.
So i guess the picture that u see contains restored information from this one filter map in conv5 layer.
Try to check other filters in your conv5 layer.

[Trekky12]

@nyf-nyf yes this is only the first filter, but I've also plotted the other 255 filters and the result is similar. There is always the gray border on the right/bottom and mostly the pattern is the same like in the image above

[nyf-nyf]

@Trekky12 Did you train the net on the images with 0-255 values and not 0-1?

Try to remove set_raw_scale and check the result

[Trekky12]

@nyf-nyf I used the reference caffenet directly (so yes the images are 0-255), but like shown above I already used

transformer.set_raw_scale('data', 255) 

[nyf-nyf]

@Trekky12 do you have different results in the conv5 blobs?
The problem is only with deconv layers? Does the net work fine?

[Trekky12]

@nyf-nyf you mean the difference between with and without set_raw_scale. With set_raw_scale the result seems to be much better, but the gray border right/bottom is in both cases

[nyf-nyf]

@Trekky12 i mean can u see the conv5 blobs to understand if the net works fine.
They must be different. One feature map can activate on some part of the cat, the other on ther other part. Some of them may be fully dark.

U can use the class bb to display them in opencv

cv2.imshow("conv5", bb(net, "conv5").get_image().astype(np.uint8))

[Trekky12]

@nyf-nyf your code produces a 4603x13 Pixel big image with different filter results, but I can't see whether this is one part of a cat or not. Is this the intended behavior?

The result of the deconvolution and unpooling produces for each of the 256 filter numbers a slightly different result, but the grey border is there always. How do you know which one of the 256 filters is the most significant?

[nyf-nyf]

@Trekky12 4603 px is too big. Try to do this. It will force the size of the window to max width and height, but will display only a part of filters.

bl_conv5 = bb(net, "conv5")
bl_conv5.display_all = False
bl_conv5.max_width = 500
bl_conv5.max_height = 500
cv2.imshow("conv5", bl_conv5.get_image().astype(np.uint8))

The gray border is ok, because the of the changing the values size to 0-255.
The question is why do you see simply the same picture on different filters.

BTW here is the function to scale the image, if the filters are very small to understand the changes between them.

def scale_image(im, factor):
    im = cv2.resize(im, (im.shape[1]*factor, im.shape[0]*factor))
    im = im.astype(np.float32)
    im -= im.min()
    im /= im.max()
    im *= 255
    return im.astype(np.uint8)

You can try to use such display
cv2.imshow("conv5", scale_image(bl_conv5.get_image().astype(np.uint8), 5))

But if u scale it with factor 5 with width 500 - then u get the image with width 2500, so set the max_width and height to lower values.

[Trekky12]

@nyf-nyf the first function is apparently not working, because the windows stays the same. Also there is a Runtime Warning:

RuntimeWarning: invalid value encountered in divide d /= d.max()

The scale_image function results in a empty image with transparent background. I don't think it is working.

I don't see the exactly same picture on different filter numbers, but most of them are nearly the same. Also a few show the following things:

But most of the images show the content of the image above, sometimes with slightly different contrast values.

Is it possible that you write a small tutorial how to use the deconvolutional/unpooling part with the help of the bvlc_reference_caffenet and the cat.jpg image?

That would be great!

[mariolew]

@Trekky12 maybe you can take a look at #3376

[qingzew]

anyone who has a full example about this?

[mariolew]

@qingzew You can take a look at #3376

[qingzew]

@mariolew thank you