Greater performance loss when training on padded MNIST

[zhuokaizhao]

Hi, first of all thank you for all the hard work you've put on this research project and make it very simple to use within PyTorch.

Recently I've been testing some models' translation/rotation equivariance properties. And I notice some unexpected behavior when testing models built with e2cnn and its non-e2cnn variant (for example, the tutorial you released https://github.com/QUVA-Lab/e2cnn/blob/master/examples/model.ipynb that works on MNIST). Since I would want to translate the image during testing, I prepared the padded-MNIST by padding each 28x28 MNIST image to 64x64 with constant zero padding to the right and bottom (so the original MNIST image is at the top left of the padded image). I trained the two networks and expected the training accuracy to be similar from both implementations. However, the non-e2cnn network performs similar to when training with the non-padded original MNIST (99% training accuracy), while the e2cnn model performs much worse (only 70% training accuracy).

It seems that the padding is the only change here. But I don't quite understand the reason. May I know if you had any insight why would padding influence the e2cnn's performance so much? Thank you very much.

[Gabri95]

Hi @zhuokaizhao

thanks for your nice words! :)

That seems a weird behaviour and to be honest I don't see how the padding could make a difference here.
If you can, could you share a minimal version of the code which produces the same issue? That could help me understand what is happening.

Best,
Gabriele

[zhuokaizhao]

Hi @Gabri95

Thank you for your prompt reply. I've made a ipython notebook that could be found here https://github.com/zhuokaizhao/e2cnn-experiment/blob/main/e2cnn-discussion.ipynb. I used local MNIST files to run the script and the notebook keeps the validation result after I ran on my machine. So maybe you could just take a look at the code and see if anything is a red flag to you. But please feel free to let me know if you had any trouble running it. Thank you.

Best regards,
Zhuokai

[Gabri95]

hey! I tried to run your code but I get a unpack requires a buffer of 8 bytes error when I try to load the dataset using loadlocal_mnist.
Did you experience anything similar?
Do you know how I could fix it?

Best,
Gabriele