quetion of Multi Query Attention

[hahadashi]

hi，have you test your model the speed model difference of multi head and muti query？

[AnShengqiang]

I was confused. I tested two types of attention and there was almost no difference in time between them.

[M-e-r-c-u-r-y]

@hahadashi no I haven't done concrete testing of speed difference between the models, those require larger sequences such as those in wmt dataset and not multi30k dataset. I tried loading the WMT14 dataset in colab and it wouldn't load so I just wrote the code for multi query attention and ran it on the smaller dataset and made this repo.
@AnShengqiang the time difference would be in inference time and generally only visible for large sequences, sadly multi30k dataset consists of smaller sequences and thus you wouldn't be able to notice the gains of multi query attention being more efficient.

[AnShengqiang]

@M-e-r-c-u-r-y Yes, I compared the predicted time, and found that the time is basically the same, and sometimes even contrary to the results of the paper.

I was wondering, how do I understand that it can only speed up with long sequences?