Training Speed is Slow

[ajaykrishnan23]

I tried training EfficientNet_B0 vs Resnet18 with the same image_size and other parameters to test their comparisons In speed. Despite Efficientnet being over 2x smaller than resnet18 with 12M parameters, it was still taking more time.

Any help here would be wonderful. Thanks in advance!

[gfotedar]

I see the same issue. I think at least part of the reason is that pytorch depth-wise convolution is very slow. See this. Try it in fp16 if you can by enabling autocast and change to latest cudnn and pytorch to get the current best optimized kernels.
I've done that and that has helped at least a bit but its still slower than resnet with same amount of parameters on fp16

[lukemelas]

Yes, as @gfotedar mentioned, PyTorch fp32 depthwise convs are quite slow. Also, since EfficientNet uses depthwise convs, it's always going to be slower than a model with the same number of parameters that does not use depthwise convs.