Separable Self Attention to other models

[parakh08]

Hi,

I was wondering if you have replaced normal attention with this separable self attention layer to other transformer networks like DeIT as well? What were your observations in it?

[sacmehta]

We tried to replace it in few architectures (Deit and Swin) and observations were consistent with the ones reported in the paper. However, separable attention needs to be carefully integrated especially if the models have relative positional biases.

[parakh08]

Yeah, that was what I was wondering. I tried to integrate it in Deit for the classification task, however, the accuracy that I am obtaining is quite off the original Deit accuracy. Will it be possible for you to share the models?

This is how the separable attention looks in my current distilled model for Deit, 224 image size and 16 patch.

[parakh08]

@sacmehta can you please help me in this?