3D Upsampling

[andevellicus]

Description

Currently MXNet does not support upsampling on 5D tensors (3D images). There is more and more work being done in the medical field where 3D images are very common. As MXNet supports 3D transposed convolutions, 3D Upsampling would be very helpful as well, since there are instances where upsampling has been shown to be as effective if not more so than a transposed convolutions.

References

PyTorch, TF both support 3D upsampling.

[leezu]

If it's supported by cuDNN or Intel oneDNN, you may only need to edit a few lines to expose the feature in MXNet when built with either of the backends.

[andevellicus]

I'd be happy to give it a go, just not sure how to go about doing that. I'm using the Julia bindings, if that makes a difference.

Also, I think there's a way of using average pooling backwards to accomplish the same thing?

[ChaiBapchya]

Great to have this use-case. @andevellicus Thanks for bringing it up. While your experience in Julie will be handy, we would still need work to be done on MXNet Backend [C/C++] because those few lines that @leezu mentioned would go somewhere here

For Upsampling Forward & Backward
https://github.com/apache/incubator-mxnet/blob/3b23c2de950fb0e4d44560f4c7ea933a520c526c/src/operator/nn/upsampling-inl.h#L100

CPU-specific implementation
For e.g. Shape currently checks for 4D input [2D image]
https://github.com/apache/incubator-mxnet/blob/eceb5f2c1c494094c1a697286f2c4560b7ca472e/src/operator/nn/upsampling.cc#L44-L45
https://github.com/apache/incubator-mxnet/blob/eceb5f2c1c494094c1a697286f2c4560b7ca472e/src/operator/nn/upsampling.cc#L61-L62

There don't seem to be GPU-specific implementations at the moment. So we are good on that.

You can take a stab at updating the forward & backward implementations. Additionally, we could add a test for this 3D image use-case
Similar to
https://github.com/apache/incubator-mxnet/blob/f1f3f44166e2e47afad6c65025fb48dd47efeb65/tests/python/gpu/test_operator_gpu.py#L1481-L1489

I can help review your PR. Feel free to ping me if you need any help / have specific doubts related to contributing code to the MXNet Backend.

[r3stl355]

I see this was created a while ago. Was it implemented? If not, I'd like to give it a try.

[ChaiBapchya]

Hey @r3stl355 I don't believe any update was made. Feel free to implement it! Thanks!

[r3stl355]

Yes, thank you for confirming, I checked and can see it's not there.

[r3stl355]

I managed to get a basic 3D sampling implemented (still only covering one straightforward case) but for that to work I had to make a modification to mshadow https://github.com/apache/incubator-mxnet/blob/da247654371664fb906ad286053c63e6fc2f5b5a/3rdparty/mshadow/mshadow/extension/spatial_upsampling_nearest.h#L52 as at the moment it scales only the last 2 dimensions whereas for 3D sampling it needs to do that for 3 last dims. I feel like I'm biting more than I can chew and would appreciate some guidance, would I have to create a separate PR to mshadow for this?

[ChaiBapchya]

As per https://github.com/dmlc/mshadow

This code base has been donated to the Apache MXNet project per #373, and repo is deprecated. Future development should continue in Apache MXNet.

Feel free to make changes to that header file, test it out on your end. You don't have to create a separate PR for it. Update it within the same one.

[r3stl355]

Nice catch, thank you. That statement about being donated only appears under DMLC repo.

[r3stl355]

I'm about to resume my work on this one. I paused my work due to job change. I've implemented forward for Nearest upsampling but came across a problem while trying to implement the same for Bilinear -the 3rd party library only supports 2D - any advise would be appreciated, I will provide more info shortly after I recreate my dev environment