pip install fails when installing pymupdf on macOS Big Sur

[martin-braun]

I want to export annotations from my PDFs on my reMarkable 2, so I'm trying to install this, but it fails:

$% sudo -H pip3 install -r ./requirements.txt
Collecting pymupdf==1.17.4
  Using cached PyMuPDF-1.17.4.tar.gz (202 kB)
Collecting shapely==1.7.0
  Using cached Shapely-1.7.0.tar.gz (349 kB)
Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.
Using legacy 'setup.py install' for shapely, since package 'wheel' is not installed.
Installing collected packages: pymupdf, shapely
    Running setup.py install for pymupdf ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/remarks/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-i49yvpm6/pymupdf/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-i49yvpm6/pymupdf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-bdzeg8ed/install-record.txt --single-version-externally-managed --compile --install-headers /opt/remarks/venv/include/site/python3.9/pymupdf
         cwd: /private/tmp/pip-install-i49yvpm6/pymupdf/
    Complete output (209 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11-x86_64-3.9
    creating build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/__init__.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/fitz.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/utils.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/__main__.py -> build/lib.macosx-11-x86_64-3.9/fitz
    running build_ext
    building 'fitz._fitz' extension
    creating build/temp.macosx-11-x86_64-3.9
    creating build/temp.macosx-11-x86_64-3.9/fitz
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include/mupdf -I/usr/local/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/opt/remarks/venv/include -I/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-11-x86_64-3.9/fitz/fitz_wrap.o
    fitz/fitz_wrap.c:4381:30: warning: expression result unused [-Wunused-value]
                if (dest->alpha) *s++;
                                 ^~~~
    fitz/fitz_wrap.c:4380:19: warning: unsequenced modification and access to 's' [-Wunsequenced]
                    *s++ = 255 - *s;
                      ^           ~
    fitz/fitz_wrap.c:4403:11: warning: assigning to 'unsigned char *' from 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
            c = PyBytes_AS_STRING(imagedata);
              ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    fitz/fitz_wrap.c:4407:11: warning: assigning to 'unsigned char *' from 'char *' converts between pointers to integer types with different sign [-Wpointer-sign]
            c = PyByteArray_AS_STRING(imagedata);
              ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    fitz/fitz_wrap.c:4867:18: warning: unused variable 'popup' [-Wunused-variable]
            pdf_obj *popup = pdf_dict_get(ctx, annot->obj, PDF_NAME(Popup));
                     ^
    fitz/fitz_wrap.c:4959:14: warning: unused variable 'name' [-Wunused-variable]
        pdf_obj *name = NULL;
                 ^
    fitz/fitz_wrap.c:5052:12: warning: unused variable 'len' [-Wunused-variable]
        size_t len = 0;
               ^
    fitz/fitz_wrap.c:5696:47: warning: passing 'char *' to parameter of type 'const unsigned char *' converts between pointers to integer types with different sign [-Wpointer-sign]
        res = fz_new_buffer_from_copied_data(ctx, data, strlen(data));
                                                  ^~~~
    /usr/local/include/mupdf/fitz/buffer.h:83:81: note: passing argument to parameter 'data' here
    fz_buffer *fz_new_buffer_from_copied_data(fz_context *ctx, const unsigned char *data, size_t size);
                                                                                    ^
    fitz/fitz_wrap.c:6024:19: warning: unused variable 'pdf' [-Wunused-variable]
        pdf_document *pdf = pdf_get_bound_document(ctx, annot->obj);
                      ^
    fitz/fitz_wrap.c:6081:16: warning: unused variable 'res' [-Wunused-variable]
        fz_buffer *res = NULL;
                   ^
    fitz/fitz_wrap.c:6080:27: warning: unused variable 'js' [-Wunused-variable]
        pdf_obj *obj = NULL, *js = NULL, *o = NULL;
                              ^
    fitz/fitz_wrap.c:6080:39: warning: unused variable 'o' [-Wunused-variable]
        pdf_obj *obj = NULL, *js = NULL, *o = NULL;
                                          ^
    fitz/fitz_wrap.c:6486:64: warning: passing 'char [3]' to parameter of type 'const unsigned char *' converts between pointers to integer types with different sign [-Wpointer-sign]
                               fz_new_buffer_from_copied_data(ctx, "  ", 1),
                                                                   ^~~~
    /usr/local/include/mupdf/fitz/buffer.h:83:81: note: passing argument to parameter 'data' here
    fz_buffer *fz_new_buffer_from_copied_data(fz_context *ctx, const unsigned char *data, size_t size);
                                                                                    ^
    fitz/fitz_wrap.c:7408:40: warning: declaration of 'struct Document' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN void delete_Document(struct Document *self){
                                           ^
    fitz/fitz_wrap.c:8030:17: warning: unused variable 'entry' [-Wunused-variable]
                int entry = 0;
                    ^
    fitz/fitz_wrap.c:8150:17: warning: unused variable 'page_n' [-Wunused-variable]
                int page_n = -1;
                    ^
    fitz/fitz_wrap.c:8380:68: warning: passing 'unsigned char *' to parameter of type 'const char *' converts between pointers to integer types with different sign [-Wpointer-sign]
                            LIST_APPEND_DROP(idlist, JM_UnicodeFromStr(hex));
                                                                       ^~~
    fitz/fitz_wrap.c:3448:41: note: passing argument to parameter 'c' here
    PyObject *JM_UnicodeFromStr(const char *c)
                                            ^
    fitz/fitz_wrap.c:8626:17: warning: unused variable 'cwlen' [-Wunused-variable]
                int cwlen = 0;
                    ^
    fitz/fitz_wrap.c:8630:36: warning: unused variable 'fb_font' [-Wunused-variable]
                fz_font *font = NULL, *fb_font= NULL;
                                       ^
    fitz/fitz_wrap.c:8627:17: warning: unused variable 'lang' [-Wunused-variable]
                int lang = 0;
                    ^
    fitz/fitz_wrap.c:8743:24: warning: unused variable 'len' [-Wunused-variable]
                Py_ssize_t len = 0;
                           ^
    fitz/fitz_wrap.c:9325:69: warning: passing 'char [3]' to parameter of type 'const unsigned char *' converts between pointers to integer types with different sign [-Wpointer-sign]
                                   fz_new_buffer_from_copied_data(gctx, "  ", 1), NULL, 0);
                                                                        ^~~~
    /usr/local/include/mupdf/fitz/buffer.h:83:81: note: passing argument to parameter 'data' here
    fz_buffer *fz_new_buffer_from_copied_data(fz_context *ctx, const unsigned char *data, size_t size);
                                                                                    ^
    fitz/fitz_wrap.c:9284:22: warning: unused variable 'page2' [-Wunused-variable]
                pdf_obj *page2 = NULL;
                         ^
    fitz/fitz_wrap.c:9361:26: warning: unused variable 'page2' [-Wunused-variable]
                    pdf_obj *page2 = pdf_lookup_page_loc(gctx, pdf, nb, &parent2, &i2);
                             ^
    fitz/fitz_wrap.c:9504:29: warning: unused variable 'seps' [-Wunused-variable]
                fz_separations *seps = NULL;
                                ^
    fitz/fitz_wrap.c:9943:104: warning: declaration of 'struct Colorspace' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN struct Pixmap *Page__makePixmap(struct Page *self,struct Document *doc,PyObject *ctm,struct Colorspace *cs,int alpha,int annots,PyObject *clip){
                                                                                                           ^
    fitz/fitz_wrap.c:10204:142: warning: declaration of 'struct Graftmap' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN PyObject *Page__showPDFpage(struct Page *self,struct Page *fz_srcpage,int overlay,PyObject *matrix,int xref,PyObject *clip,struct Graftmap *graftmap,char *_imgname){
                                                                                                                                                 ^
    fitz/fitz_wrap.c:10276:29: warning: unused variable 'seps' [-Wunused-variable]
                fz_separations *seps = NULL;
                                ^
    fitz/fitz_wrap.c:10494:35: warning: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long') [-Wsign-compare]
                        for (i = 0; i < n; i++) {
                                    ~ ^ ~
    fitz/fitz_wrap.c:10542:53: warning: declaration of 'struct Colorspace' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_0(struct Colorspace *cs,PyObject *bbox,int alpha){
                                                        ^
    fitz/fitz_wrap.c:10552:53: warning: declaration of 'struct Colorspace' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_1(struct Colorspace *cs,struct Pixmap *spix){
                                                        ^
    fitz/fitz_wrap.c:10621:53: warning: declaration of 'struct Colorspace' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_4(struct Colorspace *cs,int w,int h,PyObject *samples,int alpha){
                                                        ^
    fitz/fitz_wrap.c:10633:32: warning: comparison of integers of different signs: 'int' and 'size_t' (aka 'unsigned long') [-Wsign-compare]
                    if (stride * h != size) THROWMSG("bad samples length");
                        ~~~~~~~~~~ ^  ~~~~
    fitz/fitz_wrap.c:11683:19: warning: unused variable 'data' [-Wunused-variable]
                char *data = NULL;              // for new file content
                      ^
    fitz/fitz_wrap.c:11686:21: warning: unused variable 'size' [-Wunused-variable]
                int64_t size = 0;
                        ^
    fitz/fitz_wrap.c:12267:40: warning: declaration of 'struct Graftmap' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN void delete_Graftmap(struct Graftmap *self){
                                           ^
    fitz/fitz_wrap.c:12284:42: warning: declaration of 'struct TextWriter' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN void delete_TextWriter(struct TextWriter *self){
                                             ^
    fitz/fitz_wrap.c:12299:96: warning: declaration of 'struct Font' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN PyObject *TextWriter_append(struct TextWriter *self,PyObject *pos,char *text,struct Font *font,float fontsize,char *language,int wmode,int bidi_level){
                                                                                                   ^
    fitz/fitz_wrap.c:12362:36: warning: declaration of 'struct Font' will not be visible outside of this function [-Wvisibility]
    SWIGINTERN void delete_Font(struct Font *self){
                                       ^
    fitz/fitz_wrap.c:12739:19: warning: incompatible pointer types passing 'struct Document *' to parameter of type 'struct Document *' [-Wincompatible-pointer-types]
      delete_Document(arg1);
                      ^~~~
    fitz/fitz_wrap.c:7408:50: note: passing argument to parameter 'self' here
    SWIGINTERN void delete_Document(struct Document *self){
                                                     ^
    fitz/fitz_wrap.c:16769:63: warning: incompatible pointer types passing 'struct Colorspace *' to parameter of type 'struct Colorspace *' [-Wincompatible-pointer-types]
        result = (struct Pixmap *)Page__makePixmap(arg1,arg2,arg3,arg4,arg5,arg6,arg7);
                                                                  ^~~~
    fitz/fitz_wrap.c:9943:116: note: passing argument to parameter 'cs' here
    SWIGINTERN struct Pixmap *Page__makePixmap(struct Page *self,struct Document *doc,PyObject *ctm,struct Colorspace *cs,int alpha,int annots,PyObject *clip){
                                                                                                                       ^
    fitz/fitz_wrap.c:17217:74: warning: incompatible pointer types passing 'struct Graftmap *' to parameter of type 'struct Graftmap *' [-Wincompatible-pointer-types]
        result = (PyObject *)Page__showPDFpage(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8);
                                                                             ^~~~
    fitz/fitz_wrap.c:10204:152: note: passing argument to parameter 'graftmap' here
    SWIGINTERN PyObject *Page__showPDFpage(struct Page *self,struct Page *fz_srcpage,int overlay,PyObject *matrix,int xref,PyObject *clip,struct Graftmap *graftmap,char *_imgname){
                                                                                                                                                           ^
    fitz/fitz_wrap.c:17603:50: warning: incompatible pointer types passing 'struct Colorspace *' to parameter of type 'struct Colorspace *' [-Wincompatible-pointer-types]
        result = (struct Pixmap *)new_Pixmap__SWIG_0(arg1,arg2,arg3);
                                                     ^~~~
    fitz/fitz_wrap.c:10542:65: note: passing argument to parameter 'cs' here
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_0(struct Colorspace *cs,PyObject *bbox,int alpha){
                                                                    ^
    fitz/fitz_wrap.c:17638:50: warning: incompatible pointer types passing 'struct Colorspace *' to parameter of type 'struct Colorspace *' [-Wincompatible-pointer-types]
        result = (struct Pixmap *)new_Pixmap__SWIG_1(arg1,arg2);
                                                     ^~~~
    fitz/fitz_wrap.c:10552:65: note: passing argument to parameter 'cs' here
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_1(struct Colorspace *cs,struct Pixmap *spix){
                                                                    ^
    fitz/fitz_wrap.c:17777:50: warning: incompatible pointer types passing 'struct Colorspace *' to parameter of type 'struct Colorspace *' [-Wincompatible-pointer-types]
        result = (struct Pixmap *)new_Pixmap__SWIG_4(arg1,arg2,arg3,arg4,arg5);
                                                     ^~~~
    fitz/fitz_wrap.c:10621:65: note: passing argument to parameter 'cs' here
    SWIGINTERN struct Pixmap *new_Pixmap__SWIG_4(struct Colorspace *cs,int w,int h,PyObject *samples,int alpha){
                                                                    ^
    fitz/fitz_wrap.c:21495:19: warning: incompatible pointer types passing 'struct Graftmap *' to parameter of type 'struct Graftmap *' [-Wincompatible-pointer-types]
      delete_Graftmap(arg1);
                      ^~~~
    fitz/fitz_wrap.c:12267:50: note: passing argument to parameter 'self' here
    SWIGINTERN void delete_Graftmap(struct Graftmap *self){
                                                     ^
    fitz/fitz_wrap.c:21557:21: warning: incompatible pointer types passing 'struct TextWriter *' to parameter of type 'struct TextWriter *' [-Wincompatible-pointer-types]
      delete_TextWriter(arg1);
                        ^~~~
    fitz/fitz_wrap.c:12284:54: note: passing argument to parameter 'self' here
    SWIGINTERN void delete_TextWriter(struct TextWriter *self){
                                                         ^
    fitz/fitz_wrap.c:21678:59: warning: incompatible pointer types passing 'struct Font *' to parameter of type 'struct Font *' [-Wincompatible-pointer-types]
        result = (PyObject *)TextWriter_append(arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8);
                                                              ^~~~
    fitz/fitz_wrap.c:12299:102: note: passing argument to parameter 'font' here
    SWIGINTERN PyObject *TextWriter_append(struct TextWriter *self,PyObject *pos,char *text,struct Font *font,float fontsize,char *language,int wmode,int bidi_level){
                                                                                                         ^
    fitz/fitz_wrap.c:21817:15: warning: incompatible pointer types passing 'struct Font *' to parameter of type 'struct Font *' [-Wincompatible-pointer-types]
      delete_Font(arg1);
                  ^~~~
    fitz/fitz_wrap.c:12362:42: note: passing argument to parameter 'self' here
    SWIGINTERN void delete_Font(struct Font *self){
                                             ^
    49 warnings generated.
    clang -bundle -undefined dynamic_lookup -L/usr/local/lib -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk build/temp.macosx-11-x86_64-3.9/fitz/fitz_wrap.o -L/usr/local/lib -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -lmupdf -lmupdf-third -o build/lib.macosx-11-x86_64-3.9/fitz/_fitz.cpython-39-darwin.so
    ld: library not found for -lmupdf
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/remarks/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-i49yvpm6/pymupdf/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-i49yvpm6/pymupdf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-bdzeg8ed/install-record.txt --single-version-externally-managed --compile --install-headers /opt/remarks/venv/include/site/python3.9/pymupdf Check the logs for full command output.

I had a bunch of other installation issues before, that I fixed. First of, I had to brew install geos to get past the first issue. Then the fitz.c was missing, because apparently pymupdf needs mupdf, but according to this comment by the author of PyMuPDF it needs to the correct version (same major and minor version). Since remarks uses PyMuPDF 1.17.4 I downloaded MuPDF 1.17.0, extracted it and moved the mupdf folder in the include folder into /usr/local/include. This didn't work, so I followed this other comment by the author of PyMuPDF, so I replaced /usr/local/include/mupdf/fitz/config.h with the fitz/_config.h file from the PyMuPDF release 1.17.4.

An so, I get this error above. I'm stuck and I feel this whole process is way to complicated. It would be great to automate this installation somehow, but for now I just want this to work, any suggestions how I can get this to work?

My setup:

$% sw_vers
ProductName:	macOS
ProductVersion:	11.1
BuildVersion:	20C5048k
$% python3 -V
Python 3.9.0
$% pip3 -V
pip 20.3.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

[lucasrla]

Hello @martin-braun,

I am sorry to hear that your setup has been way too complicated. During mine, I used poetry (instead of pip) on a macOS Catalina machine and everything went by as smooth as it gets.

It seems you are using a developer beta version (11.1) of the recently released macOS Big Sur. I have no direct experience with Big Sur yet, but judging from your stack trace and my experience with previous macOS releases, my first guess is that Apple made changes related to clang that are in conflict with PyMuPDF. Have you googled about it?

Since you already reported the issue to PyMuPDF, I would wait for their response there. If their solution requires a patch in a new version of PyMuPDF, please let me know. I am happy to bump the PyMuPDF version that remarks depends on from 1.17.4 to anything newer that solves your issue.

Thanks

[martin-braun]

Hi @lucasrla,

thank you for taking time looking into my issue. I really appreciate it.

I am sorry to hear that your setup has been way too complicated. During mine, I used poetry (instead of pip) on a macOS Catalina machine and everything went by as smooth as it gets.

So in other words, on Catalina it would also install MuPDF together with PyMuPDF? I'd like to avoid Poetry, because I want to a build a bash script that does syncing, installation and exporting all in once. The real headache is not to use pip instead of poetry, it's the fact that I have to extract files from MuPDF to insert into PyMuPDF as well as the manual install of MuPDF itself. If you tell me that this is all not necessary and (even with pip) it should install without any additional steps, I'm happy, although it would mean that I have to wait.

Btw, I even tested brew install mupdf-tools. This will also install the include-files (i.e. fitz.c, etc.), yet same incompatibilities.

I have no direct experience with Big Sur yet, but judging from your stack trace and my experience with previous macOS releases, my first guess is that Apple made changes related to clang that are in conflict with PyMuPDF. Have you googled about it?

Unfortunately, my research is not that straightforward regarding this. However the dates of the latest releases are promising to support Big Sur, but 1.17.4 is too old, to be handle Big Sur abnormalities on Clang.

Since you already reported the issue to PyMuPDF, I would wait for their response there. If their solution requires a patch in a new version of PyMuPDF, please let me know. I am happy to bump the PyMuPDF version that remarks depends on from 1.17.4 to anything newer that solves your issue.

Someone tries to build it on Big Sur by his own, but there is no report back. At least it indicates that you are probably right. Thank you for taking action in case it is required to port your software to Big Sur.

Best regards!

[lucasrla]

Unfortunately it doesn't seem like using poetry would solve your issue today.

My installation went through very smoothly because poetry automatically pulled one of the wheels that PyMuPDF have made available. The wheels are pre-built and include MuPDF.

I have just stumbled upon a thread regarding pip on Big Sur that provides great insight on what might be happening with you. It is worth reading it all (it is kind of surreal).

Then, try this:

pip install --platform macosx_10_9_x86_64 --only-binary=:all: --target=/tmp pymupdf

If the above does not work (nor any other solution posted there), then we are in deeper trouble. The alternative path that PyMuPDF recommends is building it from the source. From your first post, it seems like you followed that path already (without much success).

Side note – If I may, I would say that the important meta lesson with this issue is: if you rely on developer tools, wait before doing major OS upgrades. It always takes time (months?) before things are fully sorted out (especially with Apple and Python, which don't seem to be as in sync as they should).

Best regards

PS: If the suggestion above does not work and you need to further experiment with flags, here is a nice guide by Real Python about wheels.

[lucasrla]

I just had a quick look at the PR aimed to fix the issue with Big Sur and wheels. It seems the fix has been merged to master and is part of packaging 20.5.

This commit from 2020-11-30 indicates that pip 20.3 (which is older than the 20.3.1. that you have installed) is already using packaging 20.7.

These discoveries made me a bit more pessimist about the ideas and solutions that I have shared so far... Maybe the root cause of your issue is something else. I don't really know.

Looking forward to hearing your findings with the experiments that I have proposed so far.

[lucasrla]

I just had a new idea that sounds very promising.

There is no pre-built wheel for CPython 3.9 in PyMuPDF 1.17.4. But there are pre-built wheels for CPython 3.9 since PyMuPDF 1.17.7.

That is expected since Python 3.9 came out in early October 2020 and 1.17.4 was released in July 2020.

In other words, my current hypothesis is: the actual reason pip is building PyMuPDF 1.17.4 from source in your machine is because it is not finding a wheel that suits your Python version.

If that is the culprit, it should be easy to fix. I will look into that later today.

[lucasrla]

Meanwhile, can you please edit the remarks/requirements.txt file in your local machine, @martin-braun?

From:

pymupdf==1.17.4
shapely==1.7.0

To:

pymupdf~=1.18.4
shapely~=1.7.1

And report back if you are able to install and use remarks just fine?

Thanks

[lucasrla]

Hey @martin-braun,

I just bumped PyMuPDF and Shapely versions (commit 26254dc).

Whenever you have the time, please do let me know if that solves your installation issue on Python 3.9 and macOS Big Sur.

Thanks

[martin-braun]

Hey @lucasrla,

thank you for your effort and time.

Side note – If I may, I would say that the important meta lesson with this issue is: if you rely on developer tools, wait before doing major OS upgrades. It always takes time (months?) before things are fully sorted out (especially with Apple and Python, which don't seem to be as in sync as they should).

Yes, I get that. I had reasons to update and I knew the risk that comes with it. I'm just here to report the issue. :)

I retried the installation with the HEAD of the master. It keeps failing:

Collecting pymupdf~=1.18.4
  Using cached PyMuPDF-1.18.5.tar.gz (251 kB)
Collecting shapely~=1.7.1
  Using cached Shapely-1.7.1.tar.gz (383 kB)
Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.
Using legacy 'setup.py install' for shapely, since package 'wheel' is not installed.
Installing collected packages: pymupdf, shapely
    Running setup.py install for pymupdf ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/remarks/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-00cgmjsi/pymupdf/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-00cgmjsi/pymupdf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-410btpcs/install-record.txt --single-version-externally-managed --compile --install-headers /opt/remarks/venv/include/site/python3.9/pymupdf
         cwd: /private/tmp/pip-install-00cgmjsi/pymupdf/
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11-x86_64-3.9
    creating build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/__init__.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/fitz.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/utils.py -> build/lib.macosx-11-x86_64-3.9/fitz
    copying fitz/__main__.py -> build/lib.macosx-11-x86_64-3.9/fitz
    running build_ext
    building 'fitz._fitz' extension
    creating build/temp.macosx-11-x86_64-3.9
    creating build/temp.macosx-11-x86_64-3.9/fitz
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include/mupdf -I/usr/local/include -Imupdf/thirdparty/freetype/include -I/opt/remarks/venv/include -I/usr/local/Cellar/python@3.9/3.9.1_1/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-11-x86_64-3.9/fitz/fitz_wrap.o
    fitz/fitz_wrap.c:2735:10: fatal error: 'fitz.h' file not found
    #include <fitz.h>
             ^~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/remarks/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-00cgmjsi/pymupdf/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-00cgmjsi/pymupdf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-410btpcs/install-record.txt --single-version-externally-managed --compile --install-headers /opt/remarks/venv/include/site/python3.9/pymupdf Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the '/opt/remarks/venv/bin/python3 -m pip install --upgrade pip' command.

I tried to purge the cache and update pip, but it keeps using PyMuPDF 1.18.5 and pip won't be updated, too. Then, I found out that rmapi allows me to export annotations from my rM and I don't even need to sync the files by my own (rmapi geta /my-pdf-file ., simple as that). It has few formatting issues with pen sizes, but it works right out of the box and is enough for me.

If you wish, I can retry to install it, later. But I would not consider using remarks, if I have to sync the files by my own.

Good luck with it and Happy Christmas!

[lucasrla]

Thanks for the update, @martin-braun.

Great to know that rmapi is working for you. (I am not familiar with it, and will look into it in the future.)

Re your installation issues with remarks, it seems that your pip version has been downgraded (probably because you are installing remarks in a different environment now).

In your first log, we had:

$% pip3 -V
pip 20.3.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

But your latest log states that:

WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.

You wrote that "pip won't be updated", and that's weird. Have you tried to run /opt/remarks/venv/bin/python3 -m pip install --upgrade pip as the help message there recommends?

Unfortunately, as of today, it seems that the only option for an easy installation of remarks on macOS Big Sur (i.e., getting PyMuPDF from a wheel and not building it from source) is to have pip >= 20.3 (ref pip#988).

It is a pity that I am not able to reproduce your macOS Big Sur setup. So, if you have the time to test this out, I would really appreciate it. (As you can probably tell, I really want to nail this issue down – even if only for the benefit of other users.)

Just to make sure it is clear:

1. cd into your remarks directory (apparently it is /opt/remarks now)
2. Activate your virtual environment (apparently it now lives at /opt/remarks/venv/)
3. Check if your current pip version is >= 20.3; if it is not, run /opt/remarks/venv/bin/python3 -m pip install --upgrade pip
4. Run /opt/remarks/venv/bin/python3 -m pip install -r requirements.txt
5. 🤞

Thank you in advance and all the best for you as well!

PS: I don't expect the use of PyMuPDF 1.18.5 (instead of 1.18.4) to be the offender here. Both versions should be fine.

[lucasrla]

Due to inactivity I will close this issue for now.

In any case, we now have confirmations that remarks is working fine on:

• Python 3.9.1 (conda), h/t @akelai
• macOS Big Sur, h/t @tombarys

[martin-braun]

Sorry for the late reply. I'm just too distracted by projects now and as rmapi works for me, I see no personal priority in finding a fix for it, at the moment. However, I felt obligated to help you out for future users, but now I'm glad other people confirmed it is working, so maybe something is messed up in my system or I did something wrong.

Maybe I will review my script and fix those issues to a later time, so I welcome this issue being closed, because I doubt it will lead to something. In the future I might try to fix it. In case of any trouble, I will open a new issue, then.

Thank you for everything, nonetheless.