fused_mha default to false for nets with relu^2 activation

[borg323]

No description provided.

[Menkib64]

I suspect that other networks will have NaN problems too. I suspect that all networks are fp16 unsafe because they were trained to use fp32.
I see a few potential solution. We either need to fix old networks to be fp16 safe (I don't know how feasible this is), we can add guards against infinites in softmax kernels (like you did for cuda), or we could add quantization scalers which would adjust values to avoid overflows.

[borg323]

However the cutlass code was robust so far. It may be that updating the code to use the latest upstream version is to blame. Even so, ReLU^2 is a likely source of overflows in fp16, so avoiding unsafe code still makes sense to me.

[Menkib64]

It makes sense to disable it for a network which have clear problems. I'm worried that BT4 might have a rare NaN problem. I was also thinking how to make a generic fix for overflows. I raised it here in case you might come up with a simple generic solution. I think this should be merged, if none comes up with a simple solution soon.