[CXX11ABI] torch 2.6.0-cu126 and cu124 have different exported symbols

[vadimkantorov]

🐛 Describe the bug

The symbol _ZN3c105ErrorC2ENS_14SourceLocationESs is exported in cu124's version, but missing in cu126: some nm outputs in Dao-AILab/flash-attention#1644

I understand that because of missing symbols, flash_attention has stopped working with torch 2.7. But it was a bit surprising that the exported symbols differ between cu124 and cu126 version of the same release...

Also, a question is why torch exported _ZN3c105ErrorC2ENS_14SourceLocationESs and why flash_attention depends on it...

@malfet

Versions

torch 2.6.0-cu126 and cu124

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @seemethere @malfet @osalpekar @atalman @ptrblck @eqy @jerryzh168

[malfet]

I agree this shouldn't be the case...

[vadimkantorov]

Maybe worth adding some smoke tests on what symbols actually get exported / compare between versions. Especially if now symbols should not get auto-exported to reduce symbol pollution...

I also wonder how/why flash_attention depends on _ZN3c105ErrorC2ENS_14SourceLocationESs, as it broke its operation with torch 2.7.0.

Also, xformers hid all 2.6.0-compatible installation recipes, despite the fact that vllm hasn't released 2.7.0 version yet and flash_attention seems broken with 2.7.0...

[malfet]

Maybe worth adding some smoke tests on what symbols actually get exported / compare between versions. Especially if now symbols should not get auto-exported to reduce symbol pollution...

We do have smoke tests for those in https://github.com/pytorch/pytorch/blob/main/.ci/pytorch/smoke_test/check_binary_symbols.py but they were mostly focused on non-using CXX11 ABI before it were enabled by default

I also wonder how/why flash_attention depends on _ZN3c105ErrorC2ENS_14SourceLocationESs, as it broke its operation with torch 2.7.0.

c++filt _ZN3c105ErrorC2ENS_14SourceLocationESs-> c10::Error::Error(c10::SourceLocation, std::basic_string<char, std::char_traits<char>, std::allocator<char> >) , i.e. I suspect it's the one that is raised by any TORCH_CHECK call....

[vadimkantorov]

I also wonder if this symbol can be brought back in 2.7.1 - just to make sure that flash_attention's pip-distributed pre-compiled binaries work for torch 2.7.1 out-of-the-box?..

Maybe this symbols is needed for building PyTorch C++ extensions using TORCH_CHECK?

[malfet]

Ok, I sometimes fail to read issues: this is an expected behavior for 2.6.0 and indeed associated with CXX11 ABI migration: 12.4 were build with pre-CXX11ABI and 12.6 with it, which should have been reflected in release notes

[vadimkantorov]

12.4 were build with pre-CXX11ABI and 12.6 with it

I see - just a bit strange that this happened without version bump within the same 2.6.0 release.

And if it happened during 2.6.0 release - flash_attention's own releases seem falling behind for quite some months now...

[malfet]

I.e. the symbol name in 2.6 (and all 2.7.0) are _ZN3c105ErrorC1ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE -> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)

[vadimkantorov]

So it is expected that binaries of the most C++ extensions need to upgrade and push new binaries, right?

[malfet]

@vadimkantorov just checking: are you saying you can't build flash attention from source, or their binaries are incompatible?
If former, I'll try to submit a PR (and ask @drisspg help for review) to make them respect the flag set by torch