int8 t5, really short prompts cause NaN

[Ph0rk0z]

If using int8 (blockwise) t5 with int8 model and triton implementation, the image will NaN on shorter prompts. Int8 TE with Fp8 weights will always NaN on triton. Pytorch backend appears to work correctly. It consistently reproduced with both for Fp16 and Bf16 capable GPUs.

Chroma

int8 tensorwise + quantops (triton) = NaN
int8 T5 (triton) + bobjohnson tensorwise + paragraph = normal image
int8 T5 (triton) + bobjohnson tensorwise + 1 sentence = NaN
int8 T5 (pytorch) + bobjohnson tensorwise + any = normal image
int8 T5 (pytorch) + fp8 + any = normal image
int8 T5 (triton) + fp8 + any = NaN

I made a little chart.