ONNX backend does not support BFLOAT16 inputs

[MatthieuToulemont]

Description:

I am trying to run a model with Bfloat16 inputs in ONNX but I get the following message:

“Invalid argument: load failed for model ‘FluxEasyControlKVCacheApplyTokenDrop’: version 1 is at UNAVAILABLE state: Internal: unsupported datatype TYPE_BF16 for input ‘double_blocks_k_cache’ for model ‘FluxEasyControlKVCacheApplyTokenDrop’;\n”

It's a bit frustrating because I am able to run the model outside of Triton with Bfloat16 inputs.

I am running in the 25.02 triton ngc container

[yinggeh]

Thanks for reporting. Looks like there is no TYPE_BF16 input dtype support in Onnxruntime backend. https://github.com/triton-inference-server/onnxruntime_backend/blob/59b3b9a5f7fbf900bd2bba10e4a03917a31a2e70/src/onnxruntime_utils.cc#L366-L405
We have created an internal ticket for assessment from our engineers.

[yinggeh]

Added support for BF16 in Onnxruntime backend. Will be available in Triton 26.03 release.