-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added the unpooling layer, that does unpooling operation #2561
base: master
Are you sure you want to change the base?
Conversation
…ed by Matthew Zeiler.
Hey @nyf-nyf, unfortunately there is no example for deconv networks. Can you explain the debugging with unpooling, relu and deconvolution? Thank! |
I can show you the example of the net, that I'm working with. So the mechanism is simple. Each pooling layer outputs the value and the mask. I use mask in unpooling layers to recover the image.
I'm using python and opencv. So I simply make all feature maps on "pool5" layer to zero except one. And I can see the activations of this one only. |
Sorry for the late reply. Thanks for the example net, but I don't really get how you make all feature maps on "pool5" to zero except one. Do you forward propagate an image and then do the net surgery? Can you please explain the python example? |
Just like that I make a copy of layer params |
Why is the backwards pass just a return statement? The backwards pass should be simple and similar to the forwards pass. Also, it doesn't make sense to have a pooling type. There is no "MAX" pooling -- it simply is taking in the indices from bottom[1]. |
@nyf-nyf If the conv5 and conv5t kernel size is 3, how are you generating the image in the conv1t opencv window you posted? |
@nyf-nyf thank you for your help. I tried to use your example and fine-tuned my net with the modified prototxt with the Deconvolution and UnPooling Layers. After obtaining the weights file I tried to load in with the help of the new Matlab interface. I did the following in Matlab:
I've removed the MemoryData and use the following instead:
I've also removed the accuracy and SoftmaxWithLoss Layer for the prediction. After some testing I recognized, that the weights and data of the conv*t layers are zero. How can this be avoided? Thanks, |
@Trekky12 according to the documentation mensioned at the top, the weights of Deconvolutional Layers are made of transposed weights of the same Conv layers. In python i manually apply them before starting the net training or testing like that
|
@ahaque
It is initialized with 2 params. The net and the name of the blob. After net forwarding you can show the layers blob like that: |
@seanbell i've made the backwards method just to return, because the Deconvolutional layers, as I understood, are not used in training. Just to "debug" the net. |
@nyf-nyf thank you for this hint but neither in Matlab nor in python I can create an image. The image remains zeros.
Do you know the problem? Trekky Edit: Apparently there is indeed an image generated. I tested it with my own net and the cat.jpg testimage and the bvlc_reference_caffenet.caffemodel but I don't think the output is correct. Unfortunately the filter number doesn't matter because the same image is always generated. |
@nyf-nyf Thank you! |
@Trekky12 after forwarding can u please check the values of the blob. Max and min are 0?
If they are 0, check please the conv5 blob
Are these values are 0 and 0? |
@nyf-nyf Thank you for your quick response. The max and min are not 0:
|
@Trekky12 how are you viewing the image of the blob? May be the problem is in displaying the image? Your conv1t blob layer has values from -2 to 2. You need to convert them from 0 to 255. BTW do not forget to convert the type of an array to np.uint8 like that |
@nyf-nyf I think this was the problem. I did that and the image is indeed a little bit different and seems more accurate. Do you think this is correct for the cat.jpg testimage? Why is there the gray border around the structure? |
@Trekky12 as i saw in your previous messages, you enabled only the first filter. |
@nyf-nyf yes this is only the first filter, but I've also plotted the other 255 filters and the result is similar. There is always the gray border on the right/bottom and mostly the pattern is the same like in the image above |
@Trekky12 Did you train the net on the images with 0-255 values and not 0-1? Try to remove |
@nyf-nyf I used the reference caffenet directly (so yes the images are 0-255), but like shown above I already used
|
@Trekky12 do you have different results in the conv5 blobs? |
@nyf-nyf you mean the difference between with and without set_raw_scale. With set_raw_scale the result seems to be much better, but the gray border right/bottom is in both cases |
@Trekky12 i mean can u see the conv5 blobs to understand if the net works fine. U can use the class bb to display them in opencv
|
@nyf-nyf your code produces a 4603x13 Pixel big image with different filter results, but I can't see whether this is one part of a cat or not. Is this the intended behavior? The result of the deconvolution and unpooling produces for each of the 256 filter numbers a slightly different result, but the grey border is there always. How do you know which one of the 256 filters is the most significant? |
@Trekky12 4603 px is too big. Try to do this. It will force the size of the window to max width and height, but will display only a part of filters.
The gray border is ok, because the of the changing the values size to 0-255. BTW here is the function to scale the image, if the filters are very small to understand the changes between them.
You can try to use such display But if u scale it with factor 5 with width 500 - then u get the image with width 2500, so set the max_width and height to lower values. |
@nyf-nyf the first function is apparently not working, because the windows stays the same. Also there is a Runtime Warning:
The scale_image function results in a empty image with transparent background. I don't think it is working. I don't see the exactly same picture on different filter numbers, but most of them are nearly the same. Also a few show the following things: Is it possible that you write a small tutorial how to use the deconvolutional/unpooling part with the help of the bvlc_reference_caffenet and the cat.jpg image? That would be great! |
fixing unpooling layer build problem
anyone who has a full example about this? |
@mariolew thank you |
I've added the unpooling layer. It's described in articles of using Deconvolutional Neural Networks.
http://www.matthewzeiler.com/pubs/iccv2011/iccv2011.pdf
But i'm not expert in c++, would appreciate any fixes.
The version for cuda is missing, but works fine for me with cpu.
It helps to debug a convolutional network from bottom to top using unpooling, relu and deconvolution.