Can AQT be used to calculate qk score?

[Lisennlp]

I see that the sample codes all talk about Attention block or MLP block. Can aqt int8 only be used for parts involving parameter calculation? For example, qk score calculation, score * V calculation, can these be used aqt int8?

[lukaszlew]

Yeah it can. All Einsums/DotGeneral can be quantized.
For more advanced cases (cache), one has to use QTensor and Quantizer.quant directly. We don't have an example in docs or mini-model for that at the moment.

[Lisennlp]

But strangely, when I used AQT INT8 for Q * K and score * V, it was slower.

`

eqn = "BTNH,BSNH->BNTS"

dot_general = aqt_utils.DenseGeneral(quant=self.quant)

logits = dot_general(eqn, query, key)`