Installation fails on macOS (Apple Silicon): multiple build errors and incompatibilities

[johnchase]

I'm trying to install and run BacTermFinder on macOS (Apple Silicon, M2) and have run into multiple build errors from pip install -r requirements.txt. The repository doesn't currently indicate any platform limitations, but several dependencies fail to build natively on ARM (e.g., grpcio, scikit-bio, lightgbm, etc.).

Environment

macOS 13 (Apple Silicon, M2)
Conda: miniforge3 with Python 3.9

conda create -n bactermfinder python=3.9
conda activate bactermfinder
git clone https://github.com/BioinformaticsLabAtMUN/BacTermFinder.git
cd BacTermFinder
pip install -r requirements.txt

ERROR: Failed to build installable wheels for some pyproject.toml based projects (grpcio, lightgbm, scikit-bio)

Detailed Error

  13 warnings generated.
  clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/jhch/miniforge3/envs/bactermfinder/lib -L/Users/jhch/miniforge3/envs/bactermfinder/lib -Wl,-rpath,/Users/jhch/miniforge3/envs/bactermfinder/lib -L/Users/jhch/miniforge3/envs/bactermfinder/lib build/temp.macosx-11.0-arm64-cpython-39/skbio/stats/__subsample.o -o build/lib.macosx-11.0-arm64-cpython-39/skbio/stats/__subsample.cpython-39-darwin.so
  ld: warning: duplicate -rpath '/Users/jhch/miniforge3/envs/bactermfinder/lib' ignored
  building 'skbio.alignment._ssw_wrapper' extension
  creating build/temp.macosx-11.0-arm64-cpython-39/skbio/alignment/_lib
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/jhch/miniforge3/envs/bactermfinder/include -arch arm64 -fPIC -O2 -isystem /Users/jhch/miniforge3/envs/bactermfinder/include -arch arm64 -I/Users/jhch/miniforge3/envs/bactermfinder/lib/python3.9/site-packages/numpy/core/include -I/Users/jhch/miniforge3/envs/bactermfinder/lib/python3.9/site-packages/numpy/core/include -I/Users/jhch/miniforge3/envs/bactermfinder/include/python3.9 -c skbio/alignment/_lib/ssw.c -o build/temp.macosx-11.0-arm64-cpython-39/skbio/alignment/_lib/ssw.o -Wno-error=declaration-after-statement
  In file included from skbio/alignment/_lib/ssw.c:38:
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/emmintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
     14 | #error "This header is only meant to be used on x86 and x64 architecture"
        |  ^
  In file included from skbio/alignment/_lib/ssw.c:38:
  In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/emmintrin.h:17:
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/xmmintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
     14 | #error "This header is only meant to be used on x86 and x64 architecture"
        |  ^
  In file included from skbio/alignment/_lib/ssw.c:38:
  In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/emmintrin.h:17:
  In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/xmmintrin.h:17:
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
     14 | #error "This header is only meant to be used on x86 and x64 architecture"
        |  ^
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:56:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
     56 |     return (__m64)__builtin_ia32_vec_init_v2si(__i, 0);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:130:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    130 |     return (__m64)__builtin_ia32_packsswb((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:155:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    155 |     return (__m64)__builtin_ia32_packssdw((__v2si)__m1, (__v2si)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:180:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    180 |     return (__m64)__builtin_ia32_packuswb((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:207:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    207 |     return (__m64)__builtin_ia32_punpckhbw((__v8qi)__m1, (__v8qi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:230:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    230 |     return (__m64)__builtin_ia32_punpckhwd((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:251:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    251 |     return (__m64)__builtin_ia32_punpckhdq((__v2si)__m1, (__v2si)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:278:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    278 |     return (__m64)__builtin_ia32_punpcklbw((__v8qi)__m1, (__v8qi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:301:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    301 |     return (__m64)__builtin_ia32_punpcklwd((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:322:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    322 |     return (__m64)__builtin_ia32_punpckldq((__v2si)__m1, (__v2si)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:343:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    343 |     return (__m64)__builtin_ia32_paddb((__v8qi)__m1, (__v8qi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:364:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    364 |     return (__m64)__builtin_ia32_paddw((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:385:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    385 |     return (__m64)__builtin_ia32_paddd((__v2si)__m1, (__v2si)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:409:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    409 |     return (__m64)__builtin_ia32_paddsb((__v8qi)__m1, (__v8qi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:433:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    433 |     return (__m64)__builtin_ia32_paddsw((__v4hi)__m1, (__v4hi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/include/mmintrin.h:456:12: error: invalid conversion between vector type '__m64' (vector of 1 'long long' value) and integer type 'int' of different size
    456 |     return (__m64)__builtin_ia32_paddusb((__v8qi)__m1, (__v8qi)__m2);
        |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  fatal error: too many errors emitted, stopping now [-ferror-limit=]
  20 errors generated.
  error: command '/usr/bin/clang' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for scikit-bio
Running setup.py clean for scikit-bio
Failed to build grpcio lightgbm scikit-bio
ERROR: Failed to build installable wheels for some pyproject.toml based projects (grpcio, lightgbm, scikit-bio)

Working solution

I was able to get BactTermFinder working with the following, however, I have not tested this extensively

conda create -n bactermfinder python=3.9 -y
conda activate bactermfinder

# Use conda-forge for all core dependencies
conda install -c conda-forge \
  numpy scipy pandas matplotlib seaborn biopython scikit-bio xgboost

# Install macOS-compatible versions of tensorflow and key pip-only dependencies
pip install \
  tensorflow-macos==2.8.0 \
  tensorflow-metal==0.5.0 \
  protobuf==3.20.* \
  h5py==3.7.0 \
  greenlet

Remove these from requirements

flatbuffers==20190709135844
greenlet==2.0.1
grpcio==1.47.0
hdmedians==0.14.2
lightgbm==3.2.1
protobuf==3.20
scikit-bio==0.5.6
tensorflow==2.8.0
tensorflow_io_gcs_filesystem==0.26.0
torch==1.13.1
xgboost==1.0.2

NOTE:

On Ubuntu, installation also required system-level headers:

sudo apt update
sudo apt install build-essential python3-dev

Final thought

This issue was time-consuming to debug, but the tool is valuable. Cleaning up the install experience, or even just documenting the platform-specific path would go a long way.

Let me know if you'd like a copy of the working environment.yml I used

[AminTaheri23]

Hello John,

Thanks for the detailed issue! You are right, I didn't include limitations of running on Apple Silicon. I tried on Windows 11 and Ubuntu 22 LTS. Hopefully for the next interaction we will go towards containrization and will release from there.