Cannot install executorch on M series macbook - install_requirements.sh fails

[byjlw]

🐛 Describe the bug

Try to install using the install steps. Throws an error

(.venv) jessewhite@MadMax executorch % ./install_requirements.sh        
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Requirement already satisfied: torch==2.6.0.dev20241218 in ./.venv/lib/python3.11/site-packages (2.6.0.dev20241218)
Requirement already satisfied: torchvision==0.22.0.dev20241218 in ./.venv/lib/python3.11/site-packages (0.22.0.dev20241218)
Requirement already satisfied: typing-extensions in ./.venv/lib/python3.11/site-packages (4.12.2)
Requirement already satisfied: cmake in ./.venv/lib/python3.11/site-packages (3.31.2)
Requirement already satisfied: pip>=23 in ./.venv/lib/python3.11/site-packages (24.3.1)
Requirement already satisfied: pyyaml in ./.venv/lib/python3.11/site-packages (6.0.2)
Requirement already satisfied: setuptools>=63 in ./.venv/lib/python3.11/site-packages (65.5.0)
Requirement already satisfied: tomli in ./.venv/lib/python3.11/site-packages (2.2.1)
Requirement already satisfied: wheel in ./.venv/lib/python3.11/site-packages (0.45.1)
Requirement already satisfied: zstd in ./.venv/lib/python3.11/site-packages (1.5.5.1)
Requirement already satisfied: timm==1.0.7 in ./.venv/lib/python3.11/site-packages (1.0.7)
Requirement already satisfied: torchaudio==2.6.0.dev20241218 in ./.venv/lib/python3.11/site-packages (2.6.0.dev20241218)
Requirement already satisfied: torchsr==1.0.4 in ./.venv/lib/python3.11/site-packages (1.0.4)
Requirement already satisfied: transformers==4.46.1 in ./.venv/lib/python3.11/site-packages (4.46.1)
Requirement already satisfied: filelock in ./.venv/lib/python3.11/site-packages (from torch==2.6.0.dev20241218) (3.16.1)
Requirement already satisfied: networkx in ./.venv/lib/python3.11/site-packages (from torch==2.6.0.dev20241218) (3.4.2)
Requirement already satisfied: jinja2 in ./.venv/lib/python3.11/site-packages (from torch==2.6.0.dev20241218) (3.1.4)
Requirement already satisfied: fsspec in ./.venv/lib/python3.11/site-packages (from torch==2.6.0.dev20241218) (2024.12.0)
Requirement already satisfied: sympy==1.13.1 in ./.venv/lib/python3.11/site-packages (from torch==2.6.0.dev20241218) (1.13.1)
Requirement already satisfied: numpy in ./.venv/lib/python3.11/site-packages (from torchvision==0.22.0.dev20241218) (2.2.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./.venv/lib/python3.11/site-packages (from torchvision==0.22.0.dev20241218) (11.0.0)
Requirement already satisfied: huggingface_hub in ./.venv/lib/python3.11/site-packages (from timm==1.0.7) (0.27.0)
Requirement already satisfied: safetensors in ./.venv/lib/python3.11/site-packages (from timm==1.0.7) (0.4.5)
Requirement already satisfied: packaging>=20.0 in ./.venv/lib/python3.11/site-packages (from transformers==4.46.1) (24.2)
Requirement already satisfied: regex!=2019.12.17 in ./.venv/lib/python3.11/site-packages (from transformers==4.46.1) (2024.11.6)
Requirement already satisfied: requests in ./.venv/lib/python3.11/site-packages (from transformers==4.46.1) (2.32.3)
Requirement already satisfied: tokenizers<0.21,>=0.20 in ./.venv/lib/python3.11/site-packages (from transformers==4.46.1) (0.20.3)
Requirement already satisfied: tqdm>=4.27 in ./.venv/lib/python3.11/site-packages (from transformers==4.46.1) (4.67.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./.venv/lib/python3.11/site-packages (from sympy==1.13.1->torch==2.6.0.dev20241218) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.11/site-packages (from jinja2->torch==2.6.0.dev20241218) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests->transformers==4.46.1) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.11/site-packages (from requests->transformers==4.46.1) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.11/site-packages (from requests->transformers==4.46.1) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.11/site-packages (from requests->transformers==4.46.1) (2024.12.14)
Using pip 24.3.1 from /Users/jessewhite/Documents/source/executorch/.venv/lib/python3.11/site-packages/pip (python 3.11)
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Processing /Users/jessewhite/Documents/source/executorch
  Running command Preparing metadata (pyproject.toml)
  /Users/jessewhite/Documents/source/executorch/.venv/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
    warnings.warn(msg, _BetaConfiguration)
  running dist_info
  creating /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info
  writing /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/dependency_links.txt
  writing entry points to /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/entry_points.txt
  writing requirements to /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/requires.txt
  writing top-level names to /private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/top_level.txt
  writing manifest file '/private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch.egg-info/SOURCES.txt'
  creating '/private/var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/pip-modern-metadata-fkii1mqi/executorch-0.5.0a0+a396b47.dist-info'
  Preparing metadata (pyproject.toml) ... done
Collecting expecttest (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for expecttest from https://files.pythonhosted.org/packages/27/fb/deeefea1ea549273817ca7bed3db2f39cc238a75a745a20e3651619f7335/expecttest-0.3.0-py3-none-any.whl.metadata
  Using cached expecttest-0.3.0-py3-none-any.whl.metadata (3.8 kB)
Collecting flatbuffers (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/41/f0/7e988a019bc54b2dbd0ad4182ef2d53488bb02e58694cd79d61369e85900/flatbuffers-24.3.25-py2.py3-none-any.whl.metadata
  Using cached flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)
Collecting hypothesis (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for hypothesis from https://files.pythonhosted.org/packages/5e/aa/d78b7642d9ab608c01c90ec54d2cfab817df9e40aa27c2a22f25f2c25ffa/hypothesis-6.122.5-py3-none-any.whl.metadata
  Using cached hypothesis-6.122.5-py3-none-any.whl.metadata (6.0 kB)
Requirement already satisfied: mpmath==1.3.0 in ./.venv/lib/python3.11/site-packages (from executorch==0.5.0a0+a396b47) (1.3.0)
Requirement already satisfied: packaging in ./.venv/lib/python3.11/site-packages (from executorch==0.5.0a0+a396b47) (24.2)
Collecting parameterized (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for parameterized from https://files.pythonhosted.org/packages/00/2f/804f58f0b856ab3bf21617cccf5b39206e6c4c94c2cd227bde125ea6105f/parameterized-0.9.0-py2.py3-none-any.whl.metadata
  Using cached parameterized-0.9.0-py2.py3-none-any.whl.metadata (18 kB)
Collecting pytest (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for pytest from https://files.pythonhosted.org/packages/11/92/76a1c94d3afee238333bc0a42b82935dd8f9cf8ce9e336ff87ee14d9e1cf/pytest-8.3.4-py3-none-any.whl.metadata
  Using cached pytest-8.3.4-py3-none-any.whl.metadata (7.5 kB)
Collecting pytest-xdist (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for pytest-xdist from https://files.pythonhosted.org/packages/6d/82/1d96bf03ee4c0fdc3c0cbe61470070e659ca78dc0086fb88b66c185e2449/pytest_xdist-3.6.1-py3-none-any.whl.metadata
  Using cached pytest_xdist-3.6.1-py3-none-any.whl.metadata (4.3 kB)
Requirement already satisfied: pyyaml in ./.venv/lib/python3.11/site-packages (from executorch==0.5.0a0+a396b47) (6.0.2)
Collecting ruamel.yaml (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for ruamel.yaml from https://files.pythonhosted.org/packages/73/67/8ece580cc363331d9a53055130f86b096bf16e38156e33b1d3014fffda6b/ruamel.yaml-0.18.6-py3-none-any.whl.metadata
  Using cached ruamel.yaml-0.18.6-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: sympy in ./.venv/lib/python3.11/site-packages (from executorch==0.5.0a0+a396b47) (1.13.1)
Collecting tabulate (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for tabulate from https://files.pythonhosted.org/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-any.whl.metadata
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Requirement already satisfied: typing-extensions in ./.venv/lib/python3.11/site-packages (from executorch==0.5.0a0+a396b47) (4.12.2)
  Link requires a different Python (3.11.8 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.11.8 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.11.8 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.11.8 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
  Link requires a different Python (3.11.8 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
Collecting numpy==1.23.2 (from executorch==0.5.0a0+a396b47)
  Obtaining dependency information for numpy==1.23.2 from https://files.pythonhosted.org/packages/10/8e/843caee5e70d9edb8b01dc9418edbf475200abde5299136683006ed2d58b/numpy-1.23.2-cp311-cp311-macosx_11_0_arm64.whl.metadata
  Using cached numpy-1.23.2-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.2 kB)
Collecting pandas (from executorch==0.5.0a0+a396b47)
  Using cached https://download.pytorch.org/whl/nightly/pandas-2.2.3-cp311-cp311-macosx_11_0_arm64.whl (11.3 MB)
Collecting attrs>=22.2.0 (from hypothesis->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for attrs>=22.2.0 from https://files.pythonhosted.org/packages/89/aa/ab0f7891a01eeb2d2e338ae8fecbe57fcebea1a24dbb64d45801bfab481d/attrs-24.3.0-py3-none-any.whl.metadata
  Using cached attrs-24.3.0-py3-none-any.whl.metadata (11 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for sortedcontainers<3.0.0,>=2.1.0 from https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting python-dateutil>=2.8.2 (from pandas->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for python-dateutil>=2.8.2 from https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas->executorch==0.5.0a0+a396b47)
  Using cached https://download.pytorch.org/whl/nightly/pytz-2024.2-py2.py3-none-any.whl (508 kB)
Collecting tzdata>=2022.7 (from pandas->executorch==0.5.0a0+a396b47)
  Using cached https://download.pytorch.org/whl/nightly/tzdata-2024.2-py2.py3-none-any.whl (346 kB)
Collecting iniconfig (from pytest->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for iniconfig from https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl.metadata
  Using cached iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB)
Collecting pluggy<2,>=1.5 (from pytest->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for pluggy<2,>=1.5 from https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl.metadata
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting execnet>=2.1 (from pytest-xdist->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for execnet>=2.1 from https://files.pythonhosted.org/packages/43/09/2aea36ff60d16dd8879bdb2f5b3ee0ba8d08cbbdcdfe870e695ce3784385/execnet-2.1.1-py3-none-any.whl.metadata
  Using cached execnet-2.1.1-py3-none-any.whl.metadata (2.9 kB)
Collecting ruamel.yaml.clib>=0.2.7 (from ruamel.yaml->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for ruamel.yaml.clib>=0.2.7 from https://files.pythonhosted.org/packages/fb/8f/683c6ad562f558cbc4f7c029abcd9599148c51c54b5ef0f24f2638da9fbb/ruamel.yaml.clib-0.2.12-cp311-cp311-macosx_13_0_arm64.whl.metadata
  Using cached ruamel.yaml.clib-0.2.12-cp311-cp311-macosx_13_0_arm64.whl.metadata (2.7 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->executorch==0.5.0a0+a396b47)
  Obtaining dependency information for six>=1.5 from https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl.metadata
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Using cached numpy-1.23.2-cp311-cp311-macosx_11_0_arm64.whl (13.3 MB)
Using cached expecttest-0.3.0-py3-none-any.whl (8.2 kB)
Using cached flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)
Using cached hypothesis-6.122.5-py3-none-any.whl (476 kB)
Using cached parameterized-0.9.0-py2.py3-none-any.whl (20 kB)
Using cached pytest-8.3.4-py3-none-any.whl (343 kB)
Using cached pytest_xdist-3.6.1-py3-none-any.whl (46 kB)
Using cached ruamel.yaml-0.18.6-py3-none-any.whl (117 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Using cached attrs-24.3.0-py3-none-any.whl (63 kB)
Using cached execnet-2.1.1-py3-none-any.whl (40 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached ruamel.yaml.clib-0.2.12-cp311-cp311-macosx_13_0_arm64.whl (132 kB)
Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: executorch
  Running command Building wheel for executorch (pyproject.toml)
  /Users/jessewhite/Documents/source/executorch/.venv/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
    warnings.warn(msg, _BetaConfiguration)
  running bdist_wheel
  running build
  command options for 'CustomBuild':
    build_base = pip-out
    build_purelib = pip-out/lib
    build_platlib = pip-out/lib.macosx-10.9-universal2-cpython-311
    build_lib = pip-out/lib.macosx-10.9-universal2-cpython-311
    build_scripts = pip-out/scripts-3.11
    build_temp = pip-out/temp.macosx-10.9-universal2-cpython-311
    plat_name = macosx-10.9-universal2
    compiler = None
    parallel = 15
    debug = None
    force = None
    executable = /Users/jessewhite/Documents/source/executorch/.venv/bin/python3
  creating /Users/jessewhite/Documents/source/executorch/pip-out
  creating /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311
  creating /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out
  deleting /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out/CMakeCache.txt
  cmake -S /Users/jessewhite/Documents/source/executorch -B /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out -DBUCK2= -DPYTHON_EXECUTABLE=/Users/jessewhite/Documents/source/executorch/.venv/bin/python3 -DCMAKE_PREFIX_PATH=/Users/jessewhite/Documents/source/executorch/.venv/lib/python3.11/site-packages -DCMAKE_BUILD_TYPE=Release -DEXECUTORCH_ENABLE_LOGGING=ON -DEXECUTORCH_LOG_LEVEL=Info -DCMAKE_OSX_DEPLOYMENT_TARGET=10.15 -DEXECUTORCH_SEPARATE_FLATCC_HOST_PROJECT=OFF -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON -DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON -DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON
  -- The C compiler identification is AppleClang 15.0.0.15000309
  -- The CXX compiler identification is AppleClang 15.0.0.15000309
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CMake Error at CMakeLists.txt:259 (add_subdirectory):
    The source directory

      /Users/jessewhite/Documents/source/executorch/backends/xnnpack/third-party/FXdiv

    does not contain a CMakeLists.txt file.


  CMake Error at CMakeLists.txt:297 (add_subdirectory):
    The source directory

      /Users/jessewhite/Documents/source/executorch/backends/xnnpack/third-party/cpuinfo

    does not contain a CMakeLists.txt file.


  CMake Error at CMakeLists.txt:332 (add_subdirectory):
    The source directory

      /Users/jessewhite/Documents/source/executorch/backends/xnnpack/third-party/pthreadpool

    does not contain a CMakeLists.txt file.


  -- Using python executable '/Users/jessewhite/Documents/source/executorch/.venv/bin/python3'
  -- Resolved buck2 as /Users/jessewhite/Documents/source/executorch/buck2-bin/buck2-99773fe6f7963a72ae5f7b737c02836e.
  -- Killing buck2 daemon
  '/Users/jessewhite/Documents/source/executorch/buck2-bin/buck2-99773fe6f7963a72ae5f7b737c02836e killall'
  -- executorch: Generating source lists
  -- executorch: Generating source file list /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out/executorch_srcs.cmake
  Error while generating /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out/executorch_srcs.cmake. Exit code: 1
  Output:

  Error:
  Traceback (most recent call last):
    File "/Users/jessewhite/Documents/source/executorch/build/buck_util.py", line 26, in run
      cp: subprocess.CompletedProcess = subprocess.run(
                                        ^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/Users/jessewhite/Documents/source/executorch/buck2-bin/buck2-99773fe6f7963a72ae5f7b737c02836e', 'cquery', "inputs(deps('//runtime/executor:program'))"]' returned non-zero exit status 2.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 232, in <module>
      main()
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 217, in main
      target_to_srcs[name] = sorted(target.get_sources(graph, runner, buck_args))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 121, in get_sources
      sources: set[str] = set(runner.run(["cquery", query] + buck_args))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/jessewhite/Documents/source/executorch/build/buck_util.py", line 31, in run
      raise RuntimeError(ex.stderr.decode("utf-8")) from ex
  RuntimeError: Command failed:
  Error validating working directory

  Caused by:
      Buck appears to be running in a stale working directory This will likely lead to failed or slow builds. To remediate, restart Buck2.


  CMake Error at build/Utils.cmake:216 (message):
    executorch: source list generation failed
  Call Stack (most recent call first):
    CMakeLists.txt:381 (extract_sources)


  -- Configuring incomplete, errors occurred!
  error: command '/Users/jessewhite/Documents/source/executorch/.venv/bin/cmake' failed with exit code 1
  error: subprocess-exited-with-error
  
  × Building wheel for executorch (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/jessewhite/Documents/source/executorch/.venv/bin/python3 /Users/jessewhite/Documents/source/executorch/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/tmpoldzrtdz
  cwd: /Users/jessewhite/Documents/source/executorch
  Building wheel for executorch (pyproject.toml) ... error
  ERROR: Failed building wheel for executorch
Failed to build executorch
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (executorch)
Traceback (most recent call last):
  File "/Users/jessewhite/Documents/source/executorch/./install_requirements.py", line 185, in <module>
    subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/jessewhite/Documents/source/executorch/.venv/bin/python3', '-m', 'pip', 'install', '.', '--no-build-isolation', '-v', '--extra-index-url', 'https://download.pytorch.org/whl/nightly/cpu']' returned non-zero exit status 1.

Versions

Collecting environment information...
PyTorch version: 2.6.0.dev20241218
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.6.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.31.2
Libc version: N/A

Python version: 3.11.8 (v3.11.8:db85d51d3e, Feb 6 2024, 18:02:37) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-14.6.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Max

Versions of relevant libraries:
[pip3] numpy==2.2.0
[pip3] torch==2.6.0.dev20241218
[pip3] torchaudio==2.6.0.dev20241218
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20241218
[conda] Could not collect

cc @larryliu0820 @mergennachin

[swolchok]

does git submodule update --init --recursive fix it?

[swolchok]

looks like git submodule update --init is enough for me; easy to skip that direction I guess? not sure why git clone doesn't just do it

[byjlw]

git clone https://github.com/pytorch/executorch.git --depth 1 --recurse-submodules --shallow-submodules && cd executorch

[byjlw]

I tried your suggestion and no cigar.
I tried command in my last comment and got a slightly different error

Error while generating /Users/jessewhite/Documents/source/executorch/pip-out/temp.macosx-10.9-universal2-cpython-311/cmake-out/executorch_srcs.cmake. Exit code: 1
  Output:

  Error:
  Traceback (most recent call last):
    File "/Users/jessewhite/Documents/source/executorch/build/buck_util.py", line 26, in run
      cp: subprocess.CompletedProcess = subprocess.run(
                                        ^^^^^^^^^^^^^^^
    File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/Users/jessewhite/Documents/source/executorch/buck2-bin/buck2-99773fe6f7963a72ae5f7b737c02836e', 'cquery', "inputs(deps('//runtime/executor:program'))"]' returned non-zero exit status 2.

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 232, in <module>
      main()
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 217, in main
      target_to_srcs[name] = sorted(target.get_sources(graph, runner, buck_args))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/jessewhite/Documents/source/executorch/build/extract_sources.py", line 121, in get_sources
      sources: set[str] = set(runner.run(["cquery", query] + buck_args))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/jessewhite/Documents/source/executorch/build/buck_util.py", line 31, in run
      raise RuntimeError(ex.stderr.decode("utf-8")) from ex
  RuntimeError: Command failed:
  Error validating working directory

  Caused by:
      0: Failed to stat `/Users/jessewhite/Documents/source/executorch/buck-out/v2`
      1: ENOENT: No such file or directory


  CMake Error at build/Utils.cmake:216 (message):
    executorch: source list generation failed
  Call Stack (most recent call first):
    CMakeLists.txt:381 (extract_sources)


  -- Configuring incomplete, errors occurred!
  error: command '/Library/Frameworks/Python.framework/Versions/3.11/bin/cmake' failed with exit code 1
  error: subprocess-exited-with-error
  
  × Building wheel for executorch (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Library/Frameworks/Python.framework/Versions/3.11/bin/python3 /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /var/folders/7k/2n2lk60d5n57_33yh299tt0c0000gn/T/tmpgj2kdiuh
  cwd: /Users/jessewhite/Documents/source/executorch
  Building wheel for executorch (pyproject.toml) ... error
  ERROR: Failed building wheel for executorch
Failed to build executorch
ERROR: Could not build wheels for executorch, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: pip3 install --upgrade pip
Traceback (most recent call last):
  File "/Users/jessewhite/Documents/source/executorch/./install_requirements.py", line 185, in <module>
    subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.11/bin/python3', '-m', 'pip', 'install', '.', '--no-build-isolation', '-v', '--extra-index-url', 'https://download.pytorch.org/whl/nightly/cpu']' returned non-zero exit status 1.

[swolchok]

the install steps

can you be more specific? Are you referring to https://pytorch.org/executorch/stable/getting-started-setup.html or a different doc? If so, are you using the release/0.4 branch as that page suggests, or main?

Here's my attempt to repro. My machine has buck2 on PATH, but it is a nonworking-for-purposes-of-executorch version. I've had no end of issues with conda, so I've removed it from my machine in favor of venv.

$ mkdir executorchinstall && cd executorchinstall && git clone -b release/0.4 https://github.com/pytorch/executorch.git && cd executorch
<success>
$ git submodule sync && git submodule update --init
<success>
$ python3.12 -m venv .etinstallvenv
$ source .etinstallvenv/bin/activate.fish
$ ./install_requirements.sh
<success>

I note that your error refers to /Users/jessewhite/Documents/source/executorch/buck2-bin/; after the steps above, executorch/buck2-bin doesn't exist for me. Do you perhaps have an outdated custom buck2 setup laying around? (though I note that 99773fe6f7963a72ae5f7b737c02836e is still the correct version hash for darwin aarch64)

[byjlw]

correct. I followed the step https://pytorch.org/executorch/stable/getting-started-setup.html here from a clean directory using a venv created from python3.11

[swolchok]

after the steps above, executorch/buck2-bin doesn't exist for me

I definitely have buck2-bin now, and I see that Utils.cmake sets it as the --cache_dir. I must have misread previously.

  0: Failed to stat `/Users/jessewhite/Documents/source/executorch/buck-out/v2`

This is an error from buck2. 1) is your disk full? buck2 should be creating that directory itself IIUC 2) have you tried again recently? I bumped the buck2 version earlier this week.

[byjlw]

Yeah I have plenty of space. I'm wondering if I can get to a clean state by removing all instances of buck2 from my machine

[joshkim81]

@byjlw any luck? I'm also experiencing the exact same thing

[joshkim81]

Strangely enough, I just got it to work. I was previously trying with python 3.11 and conda. Using python 3.12 with venv did the trick

[joshkim81]

Unbelievable - I tried it again a few times running the exact same commands and it won't work. All starting from a clean directory and venv

[swolchok]

It looks like buck2 is supposed to automatically recover from this happening. We should probably ask buck2 folks about this. Citation:

• error is coming from https://github.com/facebook/buck2/blame/fb6e0f835501993f24e05ed34127cdb9f426c96f/app/buck2_server/src/daemon/state.rs#L780
• https://github.com/facebook/buck2/blame/main/app/buck2_core/src/fs/cwd.rs#L88 is the actual failing call (should buck2 just conclude that if it can't stat the working directory, it's stale?)
• facebook/buck2@a4f9cf6 is the commit by @krallin that added the check; commit message documents the intent to recover