bfloat16 support, and an attempt at homogenizing model_dtype & precision

[francoishernandez]

bfloat16

X = steps

X = relative time

It seems to work relatively plug-n-play, but we might need to adapt a few things optimizer-wise:

• fusedadam does not seem supported;
• loss lags a bit behind compared to fp16/fp32

We might investigate some bf16-specific implementations, e.g. https://github.com/arogozhnikov/adamw_bfloat16

precision // model_dtype homogenization

Previously, model_dtype is used for training, with some "precision" deduced and applied depending on some other settings (optimizer), and precision is set in PredictConfig for inference. This PR proposes a factorization of precision at the common RunningConfig level, and dtype (actual dtype the model is cast to for training),is deduced with the same conditions as before.

TODOs:

• check refactoring did not break inference;
• clarify int8 specific case handling (-> done via dtype computed_field);
• investigate bf16 optimization;
• add some validation if needed (e.g. fusedadam + bf16 incompatibility)
• add some docs/FAQ page with various precision/dtype related specificities?

[francoishernandez]

93158fe enables amp for the bfloat16 case, which seems to work fine.

[francoishernandez]

TODO

• rename precision to compute_dtype
• rename dtype to storage_dtype (or model_dtype?)

[vince62s]

for xlm-roberta-xl(xxl) which are natively fp32, I added this here:

eole/eole/bin/convert/convert_HF.py

Line 861 in 166a18b

eole_safetensor[key] = eole_safetensor[key].to(torch.float16)

to convert them to fp16
I think since we can convert any kind of model (more and more are in bf16) maybe by default we can keep the original dtype but we can add a flag to force the storage in another dtype.