No support for arm64?

[Divkix]

I was looking at this project to use with roocode codebase indexing and found that arm64 support in docker images is not there. It was there in an earlier release like 0.0.38, what changed that it is no longer there?

[wujekbogdan]

Hey, I went to DockerHub to verify if arm64 images are indeed missing and what I found puzzled me because all michaelf34/infinity:${version}-cpu images are x86 images, whereas the michaelf34/infinity:${version} images (without the -cpu suffix) have the arm64 tag.

I've just pulled the most recent michaelf34/infinity:0.0.75 image and it turns out it runs on my M1 Mac, even though the image name does not have the -cpu suffix, so I don't really understand what this suffix means.

Anyway, michaelf34/infinity works on Apple Silicon, which is great.

[michaelfeil]

Builds for arm-64 platform directly were super slow / crashing github ci. Can potentially push it from a docker machine that support cross-platform builds.

[dodekapod]

@michaelfeil How to build locally Docker container for arm64 on arm64-based machine ?

[wujekbogdan]

@michaelfeil

Wait, I'm lost. The arm64 images are present, I'm using one at the moment. The only thing that confuses me is that some images have the -cpu suffix and some do not. These ones without the -cpu suffix have arm64 tag.

Could you clarify what the -cpu suffix actually means and why only these images have the arm64 tag?

---

@dodekapod

If you're on an arm64 machine and want to build an arm64 image, just run docker build . - it will build an image that matches your architecture.

If you meant building for a different architecture than your host machine, then docker buildx can do that - you provide the target architecture via the --platform flag, e.g. --platform linux/arm64.

[dodekapod]

For now the build cpu_auto works on amd64 but fails on arm64 as follows:

infinity/libs/infinity_emb $ docker buildx build -t michaelf34/infinity:latest-cpu -f Dockerfile.cpu_auto .
[+] Building 2.6s (13/28)                                                                                                                                     docker:default
 => [internal] load build definition from Dockerfile.cpu_auto                                                                                                           0.0s
 => => transferring dockerfile: 5.56kB                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                         0.2s
 => [internal] load .dockerignore                                                                                                                                       0.0s
 => => transferring context: 71B                                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                       0.0s
 => => transferring context: 7.09kB                                                                                                                                     0.0s
 => [base 1/3] FROM docker.io/library/ubuntu:22.04@sha256:4e0171b9275e12d375863f2b3ae9ce00a4c53ddda176bd55868df97ac6f21a6e                                              0.0s
 => => resolve docker.io/library/ubuntu:22.04@sha256:4e0171b9275e12d375863f2b3ae9ce00a4c53ddda176bd55868df97ac6f21a6e                                                   0.0s
 => CACHED [base 2/3] RUN apt-get update && apt-get install --no-install-recommends -y build-essential python3-dev libsndfile1 python3-venv python3-pip python3 curl    0.0s
 => CACHED [base 3/3] WORKDIR /app                                                                                                                                      0.0s
 => CACHED [builder  1/11] RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry POETRY_VERSION=1.8.4 python3 -                                     0.0s
 => CACHED [builder  2/11] RUN echo "Poetry version:" && poetry --version                                                                                               0.0s
 => CACHED [builder  3/11] COPY poetry.lock poetry.toml pyproject.toml README.md /app/                                                                                  0.0s
 => CACHED [builder  4/11] COPY requirements_install_from_poetry.sh requirements_install_from_poetry.sh                                                                 0.0s
 => CACHED [builder  5/11] RUN ./requirements_install_from_poetry.sh --no-root --without lint,test "https://download.pytorch.org/whl/cpu"                               0.0s
 => ERROR [builder  6/11] RUN poetry run python -m pip install --no-cache-dir onnxruntime-openvino                                                                      2.3s
------                                                                                                                                                                       
 > [builder  6/11] RUN poetry run python -m pip install --no-cache-dir onnxruntime-openvino:                                                                                 
2.013 ERROR: Could not find a version that satisfies the requirement onnxruntime-openvino (from versions: none)
2.014 ERROR: No matching distribution found for onnxruntime-openvino
------
Dockerfile.cpu_auto:45
--------------------
  43 |     COPY requirements_install_from_poetry.sh requirements_install_from_poetry.sh
  44 |     RUN ./requirements_install_from_poetry.sh --no-root --without lint,test "https://download.pytorch.org/whl/cpu"
  45 | >>> RUN poetry run python -m pip install --no-cache-dir onnxruntime-openvino
  46 |     
  47 |     COPY infinity_emb infinity_emb
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry run python -m pip install --no-cache-dir onnxruntime-openvino" did not complete successfully: exit code: 1

[michaelfeil]

Okay, so openvino does not work, which is expected.
Does the gpu image work?

[dodekapod]

@michaelfeil

On arm64 (rk3588) without Nvidia GPU it doesn't work :

infinity/libs/infinity_emb $ docker buildx build -t michaelf34/infinity:latest-cpu -f Dockerfile.nvidia_auto .
[+] Building 135.5s (15/24)                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile.nvidia_auto                                                                                            0.0s
 => => transferring dockerfile: 5.12kB                                                                                                                      0.0s
 => [internal] load metadata for docker.io/nvidia/cuda:12.9.0-base-ubuntu22.04                                                                              1.7s
 => [internal] load .dockerignore                                                                                                                           0.0s
 => => transferring context: 71B                                                                                                                            0.0s
 => [internal] load build context                                                                                                                           0.1s
 => => transferring context: 2.64MB                                                                                                                         0.1s
 => [base 1/4] FROM docker.io/nvidia/cuda:12.9.0-base-ubuntu22.04@sha256:499898d51d158ad11c3ae6defa7e80829b2be04951212dfa80faf99b99a37dfe                  17.4s
 => => resolve docker.io/nvidia/cuda:12.9.0-base-ubuntu22.04@sha256:499898d51d158ad11c3ae6defa7e80829b2be04951212dfa80faf99b99a37dfe                        0.0s
 => => sha256:66653552f68d7250965219e2677b20990930291e40fdaa9d50a46ef29e00f917 76.10MB / 76.10MB                                                            3.1s
 => => sha256:499898d51d158ad11c3ae6defa7e80829b2be04951212dfa80faf99b99a37dfe 743B / 743B                                                                  0.0s
 => => sha256:85a004af6b1851b3747aee8e5eb3d257867e62a08708a0d954c612a316c2d1d0 1.37kB / 1.37kB                                                              0.0s
 => => sha256:ee5b0d8711cf7b8adac639fc6ac4113380e3bdba0856f5857dc68f5169b09b1a 11.63kB / 11.63kB                                                            0.0s
 => => sha256:67b06617bd6bbb299a723709813a971289b935f40eaff66a3348adf478cd41f6 27.35MB / 27.35MB                                                            3.6s
 => => sha256:3a461e1c60dd3306241c1ac9c3a72dace50e832ea30b8b2508a468f93bd4490d 4.59MB / 4.59MB                                                              0.7s
 => => sha256:101ff83ab9a75bf1a8044f31e570427216084bb253ab844969d633518e5cf34a 181B / 181B                                                                  0.9s
 => => sha256:1bba15468fcc82586ce921fdc422649a192dcc635f587390cd01beb95ca6dfb5 6.89kB / 6.89kB                                                              1.0s
 => => extracting sha256:67b06617bd6bbb299a723709813a971289b935f40eaff66a3348adf478cd41f6                                                                   4.3s
 => => extracting sha256:3a461e1c60dd3306241c1ac9c3a72dace50e832ea30b8b2508a468f93bd4490d                                                                   0.6s
 => => extracting sha256:66653552f68d7250965219e2677b20990930291e40fdaa9d50a46ef29e00f917                                                                   8.4s
 => => extracting sha256:101ff83ab9a75bf1a8044f31e570427216084bb253ab844969d633518e5cf34a                                                                   0.0s
 => => extracting sha256:1bba15468fcc82586ce921fdc422649a192dcc635f587390cd01beb95ca6dfb5                                                                   0.0s
 => [base 2/4] RUN apt-get update && apt-get install --no-install-recommends -y build-essential python3-dev libsndfile1 python3-venv python3-pip python3   36.0s
 => [base 3/4] RUN rm -rf /usr/local/cuda-12.9/compat                                                                                                       0.3s
 => [base 4/4] WORKDIR /app                                                                                                                                 0.0s
 => [builder 1/8] RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=/opt/poetry POETRY_VERSION=1.8.4 python3 -                                 18.5s
 => [builder 2/8] RUN echo "Poetry version:" && poetry --version                                                                                            0.8s
 => [builder 3/8] COPY poetry.lock poetry.toml pyproject.toml README.md /app/                                                                               0.0s
 => [builder 4/8] RUN poetry install --no-interaction --no-ansi --no-root --extras "all" --without lint,test && poetry cache clear pypi --all              55.0s
 => [builder 5/8] COPY infinity_emb infinity_emb                                                                                                            0.0s
 => [builder 6/8] RUN poetry install --no-interaction --no-ansi  --extras "all" --without lint,test && poetry cache clear pypi --all                        3.5s
 => ERROR [builder 7/8] RUN poetry run python3 -m pip install --no-cache-dir https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_a  2.1s
------
 > [builder 7/8] RUN poetry run python3 -m pip install --no-cache-dir https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl:
1.877 ERROR: flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.
------
Dockerfile.nvidia_auto:47
--------------------
  45 |     RUN poetry install --no-interaction --no-ansi  --extras "${EXTRAS}" --without lint,test && poetry cache clear pypi --all
  46 |     # nvcc is not installed -> the following might break if the torch version or python version changes.
  47 | >>> RUN poetry run $PYTHON -m pip install --no-cache-dir https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
  48 |     
  49 |     # TODO: remove this line
--------------------
ERROR: failed to solve: process "/bin/sh -c poetry run $PYTHON -m pip install --no-cache-dir https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiFALSE-cp310-cp310-linux_x86_64.whl" did not complete successfully: exit code: 1

Is openvino required for purely cpu setting ?

[bendangelo]

Hi, is there an update for this? Can the github ci build just be restarted?