Add flash-attn #26239
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| {% set name = "flash-attn" %} | ||
| {% set version = "2.5.8" %} | ||
|
|
||
| package: | ||
| name: {{ name|lower }} | ||
| version: {{ version }} | ||
|
|
||
| source: | ||
| url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/flash_attn-{{ version }}.tar.gz | ||
| sha256: 2e5b2bcff6d5cff40d494af91ecd1eb3c5b4520a6ce7a0a8b1f9c1ed129fb402 | ||
|
|
||
| build: | ||
| number: 0 | ||
| script: {{ PYTHON }} -m pip install . -vvv --no-deps --no-build-isolation | ||
| script_env: | ||
| - FLASH_ATTENTION_FORCE_BUILD=TRUE | ||
| - FLASH_ATTENTION_SKIP_CUDA_BUILD=FALSE | ||
| - FLASH_ATTENTION_FORCE_CXX11_ABI=FALSE | ||
| - MAX_JOBS=$CPU_COUNT | ||
| skip: true # [cuda_compiler_version in (undefined, "None")] | ||
| skip: true # [not linux] | ||
|
|
||
| requirements: | ||
| build: | ||
| - {{ compiler('c') }} | ||
| - {{ compiler('cxx') }} | ||
| - {{ compiler('cuda') }} | ||
| - {{ stdlib('c') }} | ||
| - ninja | ||
| host: | ||
| - cuda-version {{ cuda_compiler_version }} | ||
| - libcublas-dev # [(cuda_compiler_version or "").startswith("12")] | ||
| - libcusolver-dev # [(cuda_compiler_version or "").startswith("12")] | ||
| - libcusparse-dev # [(cuda_compiler_version or "").startswith("12")] | ||
|
Comment on lines
+45
to
+47
There was a problem hiding this comment. Why are these deps listed if conda-build cannot detect any links to these libraries? It also doesn't look like the upstream library has any linking flags in the setup script CUDAExtension module. If they are needed, then the recipe needs a patch to switch from static to dynamic linking... but these packages don't contain the static libraries, so I'm not sure how static linking could be happening. There was a problem hiding this comment. Added these because I was getting errors like the following: [1/49] /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/nvcc -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/cutlass/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/TH -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/THC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/python3.11 -c -c /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.cu -o /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math -gencode arch=compute_80,code=sm_80 -gencode arch=compute_90,code=sm_90 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=flash_attn_2_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -ccbin /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/x86_64-conda-linux-gnu-cc
FAILED: /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.o
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/nvcc -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/cutlass/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/TH -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/THC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/python3.11 -c -c /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.cu -o /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O3 -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math -gencode arch=compute_80,code=sm_80 -gencode arch=compute_90,code=sm_90 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=flash_attn_2_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -ccbin /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/x86_64-conda-linux-gnu-cc
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
In file included from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_launch_template.h:7,
from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.cu:5:
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
6 | #include <cusparse.h>
| ^~~~~~~~~~~~
compilation terminated.
In file included from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_launch_template.h:7,
from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.cu:5:
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
6 | #include <cusparse.h>
| ^~~~~~~~~~~~
compilation terminated.
In file included from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_launch_template.h:7,
from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src/flash_bwd_hdim128_bf16_sm80.cu:5:
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
6 | #include <cusparse.h>
| ^~~~~~~~~~~~
compilation terminated.
[2/49] /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/x86_64-conda-linux-gnu-c++ -MMD -MF /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/flash_api.o.d -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fPIC -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work=/usr/local/src/conda/flash-attn-2.5.8 -fdebug-prefix-map=/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/include -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib/stubs -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib/stubs -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/include -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib/stubs -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib/stubs -fPIC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/cutlass/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/TH -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/THC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/python3.11 -c -c /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/flash_api.cpp -o /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/flash_api.o -O3 -std=c++17 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=flash_attn_2_cuda -D_GLIBCXX_USE_CXX11_ABI=1
FAILED: /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/flash_api.o
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/bin/x86_64-conda-linux-gnu-c++ -MMD -MF /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/flash_api.o.d -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fPIC -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work=/usr/local/src/conda/flash-attn-2.5.8 -fdebug-prefix-map=/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/include -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib/stubs -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib/stubs -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/include -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/targets/x86_64-linux/lib/stubs -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib -L/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/targets/x86_64-linux/lib/stubs -fPIC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/src -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/cutlass/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/TH -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/THC -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env/include -I/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include/python3.11 -c -c /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/flash_api.cpp -o /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/build/temp.linux-x86_64-cpython-311/csrc/flash_attn/flash_api.o -O3 -std=c++17 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1017"' -DTORCH_EXTENSION_NAME=flash_attn_2_cuda -D_GLIBCXX_USE_CXX11_ABI=1
In file included from /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/csrc/flash_attn/flash_api.cpp:8:
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
6 | #include <cusparse.h>
| ^~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
subprocess.run(
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '2']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/setup.py", line 311, in <module>
setup(
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/__init__.py", line 104, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/setup.py", line 266, in run
return super().run()
^^^^^^^^^^^^^
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
self.run_command("build")
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
self.run_command(cmd_name)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
super().run_command(command)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 91, in run
_build_ext.run(self)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 873, in build_extensions
build_ext.build_extensions(self)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
self._build_extensions_serial()
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
self.build_extension(ext)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 252, in build_extension
_build_ext.build_extension(self, ext)
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 560, in build_extension
objects = self.compiler.compile(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 686, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1774, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/python -u -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /tmp/pip-wheel-7hn5_c75
cwd: /home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/
Building wheel for flash_attn (setup.py): finished with status 'error'
ERROR: Failed building wheel for flash_attn
Running setup.py clean for flash_attn
Running command python setup.py clean
No CUDA runtime is found, using CUDA_HOME='/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_build_env'
error: pathspec 'csrc/cutlass' did not match any file(s) known to git
torch.__version__ = 2.1.2.post303
/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.11/site-packages/setuptools/__init__.py:81: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.fetch_build_eggs(dist.setup_requires)
running clean
removing 'build/temp.linux-x86_64-cpython-311' (and everything under it)
removing 'build/lib.linux-x86_64-cpython-311' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.11' does not exist -- can't clean it
removing 'build'
Failed to build flash_attn
ERROR: Could not build wheels for flash_attn, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
File "$PREFIX/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "$PREFIX/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "$PREFIX/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for flash_attn, which is required to install pyproject.toml-based projects
Removed build tracker: '/tmp/pip-build-tracker-vjr8_wy1'
Traceback (most recent call last):
File "/home/conda/staged-recipes-copy/.ci_support/build_all.py", line 261, in <module>
build_all(os.path.join(root_dir, "recipes"), args.arch)
File "/home/conda/staged-recipes-copy/.ci_support/build_all.py", line 151, in build_all
build_folders(recipes_dir, folders, arch, channel_urls)
File "/home/conda/staged-recipes-copy/.ci_support/build_all.py", line 207, in build_folders
conda_build.api.build([recipe], config=get_config(arch, channel_urls))
File "/opt/conda/lib/python3.10/site-packages/conda_build/api.py", line 250, in build
return build_tree(
File "/opt/conda/lib/python3.10/site-packages/conda_build/build.py", line 3762, in build_tree
packages_from_this = build(
File "/opt/conda/lib/python3.10/site-packages/conda_build/build.py", line 2634, in build
utils.check_call_env(
File "/opt/conda/lib/python3.10/site-packages/conda_build/utils.py", line 408, in check_call_env
return _func_defaulting_env_to_os_environ("call", *popenargs, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/conda_build/utils.py", line 384, in _func_defaulting_env_to_os_environ
raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/home/conda/staged-recipes/build_artifacts/flash-attn_1715034607333/work/conda_build.sh']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/home/weiji/Documents/github/staged-recipes/build-locally.py", line 101, in <module>
main()
File "/home/weiji/Documents/github/staged-recipes/build-locally.py", line 95, in main
run_docker_build(ns)
File "/home/weiji/Documents/github/staged-recipes/build-locally.py", line 33, in run_docker_build
subprocess.check_call([script])
File "/home/weiji/mambaforge/envs/condalock/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.scripts/run_docker_build.sh']' returned non-zero exit status 1.There was a problem hiding this comment. Oh yep, we're getting that same There was a problem hiding this comment. Ah! It's because Pytorch's ATen includes the cusparse header in its headers. OK. Then we need to list these deps in both There was a problem hiding this comment. Ok, added |
||
| - packaging | ||
| - pip | ||
| - psutil | ||
| - python | ||
| - pytorch | ||
| - pytorch =*=cuda* | ||
| - libtorch # required until pytorch run_exports libtorch | ||
| - setuptools | ||
| - wheel | ||
| run: | ||
| - pytorch =*=cuda* | ||
| - python | ||
| - einops | ||
|
|
||
| test: | ||
| imports: | ||
| - flash_attn | ||
|
There was a problem hiding this comment. This test may need to be commented out because the test runners don't have a GPU, so imports might fail. There was a problem hiding this comment. Imports seemed to have worked at https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=928722&view=logs&j=4f860608-e5f8-5c9c-4eb0-308a99ecb61e&t=02ef1a5c-d960-5c54-fcea-983775f057bb&l=1352 done
export PREFIX=/home/conda/staged-recipes/build_artifacts/flash-attn_1715047997366/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
export SRC_DIR=/home/conda/staged-recipes/build_artifacts/flash-attn_1715047997366/test_tmp
import: 'flash_attn'
import: 'flash_attn'
+ pip check
No broken requirements found.
+ exit 0 |
||
| commands: | ||
| - pip check | ||
| requires: | ||
| - pip | ||
carterbox marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| about: | ||
| home: https://github.com/Dao-AILab/flash-attention | ||
| summary: 'Flash Attention: Fast and Memory-Efficient Exact Attention' | ||
| license: BSD-3-Clause | ||
| license_file: LICENSE | ||
weiji14 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| extra: | ||
| recipe-maintainers: | ||
| - weiji14 | ||
carterbox marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you don't set
FLASH_ATTENTION_FORCE_BUILD=TRUE, the package tries to download pre-build binaries instead of building them. Pre-built binaries are not allowed on our channel, you must compile all binaries with our toolchains.