Please use the new API settings to control TF32 behavior, ...

[wasertech]

System Info

UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)

I'm having an issue with vllm and this warning seams related to transformers? You can find the corresponding error at vllm#29349 with relevant info about it.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

docker run --runtime nvidia --gpus=all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 vllm/vllm-openai:v0.11.2 --model swiss-ai/Apertus-8B-Instruct-2509

Expected behavior

The new PyTorch API should be used so that MoE models can be loaded and inferred on the Turing architecture.

[Rocketknight1]

Hmm - this API is unsupported below Torch 2.9, but the old API now throws warnings on Torch 2.9 and above. I think the only solution is to add a version check in our code wherever we call these, and then use that to choose the right API. If anyone is willing to make that PR and test it on older/newer versions of Torch, feel free!

[haroon712]

I would like to work on this issue.

[khushali9]

@Rocketknight1 I am new to transformers git, have been reading code and learning on my own as I can make contributions. I wanted to ask how can I reach out to contributors (any discord,I did join one but was hard to find dev team) of the code, or if I have questions, shall I reach out in PR or issue discussion ?

[AnshArya927]

Hi team,

I would like to work on this issue and contribute a fix. I have experience with PyTorch, Docker, and HuggingFace Transformers, and I’m familiar with the new TF32 API (torch.backends.cudnn.conv.fp32_precision and torch.backends.cuda.matmul.fp32_precision).

I plan to:

• Update vLLM code to use the new PyTorch TF32 API for MoE models
• Ensure backward compatibility for older PyTorch versions
• Test with Turing/RTX GPUs using Docker

Please assign this issue to me so I can submit a PR with a fix.

Thank you!

[wasertech]

... test it on older/newer versions of Torch, ...

I agree with @Rocketknight1 a test on the latest known stable version of pytorch would help catch those changes in the API ahead of time.

[khushali9]

... test it on older/newer versions of Torch, ...

I agree with @Rocketknight1 a test on the latest known stable version of pytorch would help catch those changes in the API ahead of time.

@wasertech I have my PR out and made unit test and tested on older/newer version added in different environment. I only had to make another PR cause my first PR got little messy with make style. I have fixed it in second PR and removed tests cause it will anyways not work on CI cause we would not have all versions of pytorch.

[wasertech]

Thank you for looking into it @khushali9 I am glad the new torch API for FT32 is now supported, will give it a try when I have some free time.

[khushali9]

Sure @wasertech that will be great.