Unable to install under M1 Macintosh mode (works in x86/rosetta 2 mode)

[bschollnick]

I am unable to install pymupdf under Mac OS X Big Sur Native M1 (eg pip install PyMuPDF), whereas in Intel mode (Rosetta 2/aka Intel translation mode) it works without any issue.

(quickbbs) benjamin@Nerv quickbbs % pip install pymupdf
Collecting pymupdf
  Using cached PyMuPDF-1.18.6.tar.gz (258 kB)
Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.
Installing collected packages: pymupdf
    Running setup.py install for pymupdf ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf
         cwd: /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11-arm64-3.9
    creating build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/__init__.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/fitz.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/utils.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/__main__.py -> build/lib.macosx-11-arm64-3.9/fitz
    running build_ext
    building 'fitz._fitz' extension
    creating build/temp.macosx-11-arm64-3.9
    creating build/temp.macosx-11-arm64-3.9/fitz
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/opt/openssl@1.1/include -I/usr/local/include/mupdf -I/usr/local/include -Imupdf/thirdparty/freetype/include -I/opt/homebrew/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/tcl-tk/include -I/Users/benjamin/venvs/quickbbs/include -I/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-11-arm64-3.9/fitz/fitz_wrap.o
    fitz/fitz_wrap.c:2767:10: fatal error: 'fitz.h' file not found
    #include <fitz.h>
             ^~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf Check the logs for full command output.

It appears that it is being built and the wheel is being ignored.

Your configuration (mandatory)

• MAc OS X Big Sur, M1 Mac Mini w/16 Gb Ram

• Python 3.9.1 (default, Jan 8 2021, 12:11:08)
  [Clang 12.0.0 (clang-1200.0.32.28)] on darwin

• PyMuPDF Using cached PyMuPDF-1.18.6.tar.gz (258 kB)

Additional context (optional)

I suspect that the Universal 2 wheel just needs to be created. Cryptography, and Pillow within the last day, just started to work in M1 Native mode, due to Wheel support being added to the build systems...

If there is anything that I can assist with or help, please feel free to let me know. I'd be happy to assist.

[JorjMcKie]

It pip tries to install from sources like it does here, then the platform is not supported by any of the existing wheels.
Is that a 32bit Python?
Ah no, it seems an ARM64 architecture on Mac.
Well, this is not supportable by the wheel generation serivice I am using (Travis CI). It also isn't by the other services.
So it seems you have to generate from sources.
And that means install MuPDF first (missing this leads to fitz.h not found), then PyMuPDF as described on the repo home page ...
Hope you can make your way - I won't be able to help here.

[bschollnick]

If I am understanding what little documentation I have looked at regarding Travis CI.

Xcode 12.0 is available by adding osx_image: xcode12u to your .travis.yml.
Xcode 12.2 is available by adding osx_image: xcode12.2 to your .travis.yml.

And Xcode 12/12.2 support is what is needed for Universal support for M1 ARM... I believe. But I don't know if the travis.yml file is at Github's side, or a feature you directly control?

Could you check and see if that's available, I would greatly appreciate it.

[JorjMcKie]

I can check what happens when I add that osx_image statement to myyml file.
There currently is none, so xcode 9.4.1 is taken. Maybe by cibuildwheel ... let me check.

[JorjMcKie]

Ok, made these changes to my ravis yml:

It worked - here is the the run log from Travis CI, which shows xcode 12.2 is indeed used.
OSX-runlog.zip
The generated wheels are here: https://github.com/pymupdf/PyMuPDF-wheels/tree/osx

They have the platform tag ...-macosx_10_9_x86_64.whl - which is not any different from before. I know that pip determines wheel compatibility by simply looking at the wheel filename / platform tag, so I am afraid it will not react any differently 😟.
If that concern confirms ... I don't know.

• rename the wheel file?
• manually unzip and copy into Python folder?
• convert wheel back to egg format?

[bschollnick]

1. I tested in Rosetta2 (aka Universal 2), and it still loads fine, and works.
2. It indeed does fail under M1 Native. Renaming the Wheel was necessary, but if imported, complains about the wrong architecture (eg. X86 vs Arm64).

So just adding the compile tag for Travis CI obviously isn't the whole story...

I did some more digging, and I wish to stress, this is the first time I've even looked at Travis, beyond seeing travis logs from builds in a few repositories...

I know that it does work after being updated, because I was testing this for Pillow, and psycopg2. My understanding is that they updated their Travis CI builds to add ARM64 support.

This is a travis ci build YAML from Microsoft (LWJGL/lwjgl3#601 (comment)),
https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/.travis.yml#L305

The major differences that I see are the cmake parameters (eg -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 vs -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0) and the addition of the cmake architectures.

I'm just surprised that I can't find any clear cut documentation on how to add this to Travis CI....

The appveyor.yml also was updated in that same commit to add "PLATFORM: ARM64", so maybe that also needs to be updated? https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/appveyor.yml#L19

I just want to stress, I'll be happy to continue testing as needed.

# -----------------
  - name: "macOS x64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
    - brew install awscli
script:
    - mkdir build
    - cd build
    - "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 .."
    - cmake --build .
    - strip -u -r src/libglfw.dylib
    - "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
    - "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
    - "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
# -----------------
  - name: "macOS arm64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
    - brew install awscli
script:
    - mkdir build
    - cd build
    - "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 -DCMAKE_OSX_ARCHITECTURES=arm64 .."
    - cmake --build .
    - strip -u -r src/libglfw.dylib
    - "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
    - "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"
    - "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"

[JorjMcKie]

I am a little lost at the moment.
The examples you mentioned do not use cibuildwheel like PyMuPDF. cibuildwheel has a set of supported target architectures platform tags - see here.

Reading some info about Rosetta2 I learned that this is some feature that supports x86 apps made for Intel processors to run on M1. It is an emulation.
I do not even know whether MuPDF can run / be generated on / for x64 M1 ...

[bschollnick]

It works fine under Rosetta 2, which I’d classify more of a translation than emulation. But I’d be very surprised to see if it couldn’t be compiled under ARM, simply because a large amount of Android systems are ARM based… Not that I can read it all, but this appears to be someone that says they’ve compiled it under arm. https://www.cnblogs.com/pang1567/p/4083740.html <https://www.cnblogs.com/pang1567/p/4083740.html> I’ll dig a bit, and maybe try to compile it myself…. - Benjamin

…

On Jan 25, 2021, at 8:21 AM, Jorj X. McKie ***@***.***> wrote: I am a little lost at the moment. The examples you mentioned do not use cibuildwheel <https://github.com/joerick/cibuildwheel> like PyMuPDF. cibuildwheel has a set of supported target architectures platform tags - see here <https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip>. Reading some info about Rosetta2 I learned that this is some feature that supports x86 apps made for Intel processors to run on M1. It is an emulation. I do not even know whether MuPDF can run / be generated on / for x64 M1 ... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHPEZWK2FQX2XVMZZFZR6LS3VV5NANCNFSM4WERQXZA>.

[bschollnick]

Not sure if this helps, but from cibuildwheel: 1.8.0 22 January 2021 🌟 Added support for emulated builds! You can now build manylinux wheels on ARM64aarch64, as well as ppc64le and 's390x'. To build under emulation, register QEMU via binfmt_misc and set theCIBW_ARCHS_LINUX <https://cibuildwheel.readthedocs.io/en/latest/options/#archs> option to the architectures you want to run. See this  <https://cibuildwheel.readthedocs.io/en/latest/faq/#emulation>FAQ entry  <https://cibuildwheel.readthedocs.io/en/latest/faq/#emulation>FAQ entry <https://cibuildwheel.readthedocs.io/en/latest/faq/#emulation> for more information. (#482) According to: pypa/cibuildwheel#473 <pypa/cibuildwheel#473> cibuildwheel gains an --arch option, which is set to native by default, building 32 & 64 bit where appropriate If you include universal2 or arm64 in the arch list, then cibuildwheel downloads the new Python 3.9 (and 3.8 eventually, I think?) instead of the old one. Or this could toggle based on OS version or compiler version, but I think having it toggle based on command line options is better? If the new Python is downloaded, then the trick linked previously is used to force x86_64 or arm64 as specified. The main issue I see, which echo’s my issues earlier, is to insure you have the latest PIP, packaging, wheel modules. They all have upgrades need to support the M1. But it appears that cibuildwheel may still be working on ARM M1 support? - Benjamin

…

On Jan 25, 2021, at 8:21 AM, Jorj X. McKie ***@***.***> wrote: I am a little lost at the moment. The examples you mentioned do not use cibuildwheel <https://github.com/joerick/cibuildwheel> like PyMuPDF. cibuildwheel has a set of supported target architectures platform tags - see here <https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip>. Reading some info about Rosetta2 I learned that this is some feature that supports x86 apps made for Intel processors to run on M1. It is an emulation. I do not even know whether MuPDF can run / be generated on / for x64 M1 ... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHPEZWK2FQX2XVMZZFZR6LS3VV5NANCNFSM4WERQXZA>.

[JorjMcKie]

Yeah, I have read this: register QEMU via binfmt_misc but couldn't figure out what I have to do. Just setting the option CIBW_ARCHS_LINUX to some value is not enough definitely.

[JorjMcKie]

If the new Python is downloaded, then the trick linked previously is used to force x86_64 or arm64 as specified.
The main issue I see, which echo’s my issues earlier, is to insure you have the latest PIP, packaging, wheel modules. They all have upgrades need to support the M1.

All this is covered in one go, when the wheel generation requets is sent to Travic CI. The numerous options and "export" variables, like the one mentioned above, just control the amount of different wheels generated.
This includes taking care of the different Pythons with their latest PIPs and other stuff being installed when the Travis virtual machines are being activated.