[Bug]: Please use the new API settings to control TF32 behavior... #29349
Description
Your current environment
I'm using docker btw... this is the host (yes it has vllm but it doesn't work - I know the issue has been fixed in v0.11.1)
The output of python collect_env.py
Collecting environment information...
==============================
System Info
==============================
OS : Manjaro Linux (x86_64)
GCC version : (GCC) 15.2.1 20250813
Clang version : 20.1.8
CMake version : version 4.1.1
Libc version : glibc-2.42
==============================
PyTorch Info
==============================
PyTorch version : 2.8.0
Is debug build : False
CUDA used to build PyTorch : 12.9
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.13.7 (main, Aug 15 2025, 12:34:02) [GCC 15.2.1 20250813] (64-bit runtime)
Python platform : Linux-6.16.8-1-MANJARO-x86_64-with-glibc2.42
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.9.86
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA TITAN RTX
GPU 1: NVIDIA TITAN RTX
Nvidia driver version : 580.82.09
cuDNN version : Probably one of the following:
/usr/lib/libcudnn.so.9.11.0
/usr/lib/libcudnn_adv.so.9.11.0
/usr/lib/libcudnn_cnn.so.9.11.0
/usr/lib/libcudnn_engines_precompiled.so.9.11.0
/usr/lib/libcudnn_engines_runtime_compiled.so.9.11.0
/usr/lib/libcudnn_graph.so.9.11.0
/usr/lib/libcudnn_heuristic.so.9.11.0
/usr/lib/libcudnn_ops.so.9.11.0
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture : x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Tailles des adresses: 43 bits physical, 48 bits virtual
Boutisme : Little Endian
Processeur(s) : 24
Liste de processeur(s) en ligne : 0-23
Identifiant constructeur : AuthenticAMD
Nom de modèle : AMD Ryzen Threadripper 2920X 12-Core Processor
Famille de processeur : 23
Modèle : 8
Thread(s) par cœur : 2
Cœur(s) par socket : 12
Socket(s) : 1
Révision : 2
Accroissement de fréquence : activé
multiplication des MHz du/des CPU(s) : 71%
Vitesse maximale du processeur en MHz : 3500.0000
Vitesse minimale du processeur en MHz : 2200.0000
BogoMIPS : 6985.96
Drapeaux : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sev sev_es
Virtualisation : AMD-V
Cache L1d : 384 KiB (12 instances)
Cache L1i : 768 KiB (12 instances)
Cache L2 : 6 MiB (12 instances)
Cache L3 : 32 MiB (4 instances)
Nœud(s) NUMA : 2
Nœud NUMA 0 de processeur(s) : 0-5,12-17
Nœud NUMA 1 de processeur(s) : 6-11,18-23
Vulnérabilité Gather data sampling : Not affected
Vulnérabilité Ghostwrite : Not affected
Vulnérabilité Indirect target selection : Not affected
Vulnérabilité Itlb multihit : Not affected
Vulnérabilité L1tf : Not affected
Vulnérabilité Mds : Not affected
Vulnérabilité Meltdown : Not affected
Vulnérabilité Mmio stale data : Not affected
Vulnérabilité Old microcode : Not affected
Vulnérabilité Reg file data sampling : Not affected
Vulnérabilité Retbleed : Mitigation; untrained return thunk; SMT vulnerable
Vulnérabilité Spec rstack overflow : Mitigation; Safe RET
Vulnérabilité Spec store bypass : Mitigation; Speculative Store Bypass disabled via prctl
Vulnérabilité Spectre v1 : Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnérabilité Spectre v2 : Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnérabilité Srbds : Not affected
Vulnérabilité Tsa : Not affected
Vulnérabilité Tsx async abort : Not affected
Vulnérabilité Vmscape : Mitigation; IBPB before exit to userspace
==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.4.0
[pip3] mypy==1.18.2
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.3.3
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] onnxruntime==1.23.2
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchvision==0.23.0a0
[pip3] transformers==4.57.1
[pip3] triton==3.4.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.11.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS 0-5,12-17 0 N/A
GPU1 SYS X 6-11,18-23 1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
LD_LIBRARY_PATH=/opt/cuda/targets/x86_64-linux/lib/
CUDA_HOME=/opt/cuda
CUDA_HOME=/opt/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251124 21:45:34.047270 140368357788608 SymmetricMemory.cpp:41] Destroying Symmetric Memory Allocators
That's the host but I generally use docker so the above info is kinda irrelevant appart from hardware info.
On host I have v0.11.0 which I know doesn't work (because of the above issue I mentioned) so I tried v0.11.2 and the nightly build 0.11.2.dev201+g55c21c883; both failed with the same error.
🐛 Describe the bug
It seems related to this warning but it's unclear to me which library should address it (probably transformers) but since it happens inside the vllm docker image, it's an issue for vllm either way.
UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
EDIT: Created a ticket on hf/transformers