Unable to build/install on macOS Big Sur 11.0.1

[simmel]

What went wrong?

Not possible to install pip3 install gssapi nor build via ./setup.py build on macOS Big Sur 11.0.1:

$ pip3 install ~/src/python-gssapi
Processing /Users/simlu/src/python-gssapi
    ERROR: Command errored out with exit status 1:
     command: /Users/simlu/src/sadkfjaksdlf/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-req-build-m5_l08gj/setup.py'"'"'; __file__='"'"'/private/tmp/pip-req-build-m5_l08gj/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/tmp/pip-pip-egg-info-skp7_31t
         cwd: /private/tmp/pip-req-build-m5_l08gj/
    Complete output (6 lines):
    Cython not found, building from C files...
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/tmp/pip-req-build-m5_l08gj/setup.py", line 196, in <module>
        raise Exception("Could not find main GSSAPI shared library.  Please "
    Exception: Could not find main GSSAPI shared library.  Please try setting GSSAPI_MAIN_LIB yourself or setting GSSAPI_SUPPORT_DETECT to 'false'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '/Users/simlu/src/sadkfjaksdlf/bin/python3 -m pip install --upgrade pip' command.

$ python3 ./setup.py build
Cython not found, building from C files...
Traceback (most recent call last):
  File "/Users/simlu/src/python-gssapi/./setup.py", line 196, in <module>
    raise Exception("Could not find main GSSAPI shared library.  Please "
Exception: Could not find main GSSAPI shared library.  Please try setting GSSAPI_MAIN_LIB yourself or setting GSSAPI_SUPPORT_DETECT to 'false'

How do we reproduce?

Build on Big Sur?

Component versions (python-gssapi, Kerberos, OS / distro, etc.)

kaka:~$ kinit --version
kinit (Heimdal 1.5.1apple1)
Copyright 1995-2011 Kungliga Tekniska Högskolan
Send bug-reports to heimdal-bugs@h5l.org
kaka:~$ sw_vers
ProductName:    macOS
ProductVersion: 11.0.1
BuildVersion:   20B29
kaka:~$

[simmel]

I've build things against Kerberos libs before but I'm not sure how the whole Frameworks-thing and building C-code for Python works, that's new for me.

[frozencemetery]

Can you provide the output of krb5-config --prefix gssapi and krb5-config --libs gssapi?

I'm not sure what's changed for the new macOS - perhaps @jborean93 knows?

[jborean93]

I'm updating my Macbook today to Big Sur so I can test it out. We are seeing that in our CI tests when trying to upgrade our macOS instances to Big Sur so hopefully it's a simple solution.

[jborean93]

Ok brace yourselves because this is going to be a bumpy ride 👍

The error from setup.py stems from this exception that occurs when main_lib is None. main_lib is defined by ctypes.util.find_library('GSS') but it can also be manually set with the GSSAPI_MAIN_LIB env var.

In the case of Big Sur I found that ctypes.util.find_library is failing to find the GSS framework compared to older macOS versions

>>> platform.mac_ver()
('10.15.6', ('', '', ''), 'x86_64')
>>> ctypes.util.find_library('GSS')
'/System/Library/Frameworks/GSS.framework/GSS'
>>>

>>> platform.mac_ver()
('10.16', ('', '', ''), 'x86_64')
>>> ctypes.util.find_library('GSS')
>>>

The root of this issue stems from a new feature (Under Kernel) on Big Sur

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

This means that the actual library file at /System/Library/Frameworks/GSS.framework/GSS does not exist and lo and behold that is true!

$ ls -al /System/Library/Frameworks/GSS.framework/
total 0
drwxr-xr-x    5 root  wheel   160 Jan  1  2020 .
drwxr-xr-x  202 root  wheel  6464 Jan  1  2020 ..
drwxr-xr-x    3 root  wheel    96 Jan  1  2020 Helpers
lrwxr-xr-x    1 root  wheel    26 Jan  1  2020 Resources -> Versions/Current/Resources
drwxr-xr-x    4 root  wheel   128 Jan  1  2020 Versions

This is an issue because on all current releases of Python (3.9.1, 3.8.7, 3.7.9, 3.6.12) the find_library also checks if the path exists. Looking at the CPython code we can see that it has been fixed (alongside other Big Sur problems) with python/cpython#22855 https://github.com/python/cpython/blob/6bd94de168b58ac9358277ed6f200490ab26c174/Lib/ctypes/macholib/dyld.py#L121-L139. It still does the path check but if it doesn't exist it will check the dylib cache using an API exposed by Apple which does work.

The builtin version of Python provided by Apple has been patched so the find_library call works and the Homebrew formula for Python 3.9 (and I think 3.8) has manually backported that PR so they also work. The issue really only occurs for people who have installed Python from the python.org pkg or have manually compiled it from source themselves. I believe 3.9.2 and potentially 3.8.8 will include this fix but I'm unsure if the Python devs will backport this to 3.7 or 3.6. It seems like we have 2 ways we can go to solve this problem:

• Do nothing
  • This will be fixed in future Python versions, not sure if 3.7 or older will be fixed though
  • The brew and builtin Python have already been patched so are not affected
  • People can manually set GSSAPI_MAIN_LIB=/System/Library/Frameworks/GSS.framework/GSS when running the install
  • I'm not really a fan of this option as the alternative isn't too bad in my eyes
• Manually hardcode the path if find_library returns None
  • Still uses the nice automatic path for older versions and patched version of Python
  • This path should be static for a system library, even if it does change it will be way in the future after find_library has been patched in most working versions

I prefer the 2nd option myself because it will only use the fallback if the framework could be found and if our hardcoded path doesn't work then

python-gssapi/setup.py

Line 200 in 5ee095e

GSSAPI_LIB = ctypes.CDLL(os.path.join(main_path, main_lib))

will fail with:

>>> lib = ctypes.CDLL('/System/Library/Frameworks/GSS.framework/GSS.fake')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/System/Library/Frameworks/GSS.framework/GSS.fake, 6): image not found

I'll probably open a PR with this change sometime soon.

[jborean93]

Here is my proposed fix #236.

[simmel]

This means that the actual library file at /System/Library/Frameworks/GSS.framework/GSS does not exist and lo and behold that is true!

That's how far I got in the rabbit hole until I gave up = )

The builtin version of Python provided by Apple has been patched so the find_library call works

Except that it doesn't. Or at least some other problem arises:

$ /usr/bin/python3 setup.py build
Cython not found, building from C files...
Traceback (most recent call last):
  File "setup.py", line 350, in <module>
    main_file('misc'),
  File "setup.py", line 272, in main_file
    return make_extension('gssapi.raw.%s', module)
  File "setup.py", line 258, in make_extension
    raise OSError(source)
OSError: gssapi/raw/misc.c

Doesn't work with Homebrew Python either:

$ which python3
/usr/local/bin/python3
$ python3 --version
Python 3.9.0
$ env GSSAPI_MAIN_LIB=/System/Library/Frameworks/GSS.framework/GSS python3 setup.py build
Cython not found, building from C files...
Traceback (most recent call last):
  File "/Users/simlu/src/python-gssapi/setup.py", line 350, in <module>
    main_file('misc'),
  File "/Users/simlu/src/python-gssapi/setup.py", line 272, in main_file
    return make_extension('gssapi.raw.%s', module)
  File "/Users/simlu/src/python-gssapi/setup.py", line 258, in make_extension
    raise OSError(source)
OSError: gssapi/raw/misc.c
$

Works with Python installed via asdf though:

$ python3 --version
Python 3.7.9
$ env GSSAPI_MAIN_LIB=/System/Library/Frameworks/GSS.framework/GSS python3 setup.py build
Building from Cython files...
Skipping the s4u extension because it is not supported by your GSSAPI implementation...
Skipping the cred_store extension because it is not supported by your GSSAPI implementation...
Skipping the rfc4178 extension because it is not supported by your GSSAPI implementation...
Skipping the rfc5588 extension because it is not supported by your GSSAPI implementation...
Skipping the dce extension because it is not supported by your GSSAPI implementation...
Skipping the iov_mic extension because it is not supported by your GSSAPI implementation...
Skipping the rfc6680 extension because it is not supported by your GSSAPI implementation...
Skipping the rfc6680_comp_oid extension because it is not supported by your GSSAPI implementation...
Skipping the password_add extension because it is not supported by your GSSAPI implementation...
running build
Compiling gssapi/raw/misc.pyx because it changed.
Compiling gssapi/raw/exceptions.pyx because it changed.
[…]
building 'gssapi.raw.mech_krb5' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Application
s/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -Igssapi/raw -I./gssapi/raw -I/Users/simlu/.asdf/installs/python/3.7.9/include/python3.7m -c gssapi/raw/mech_krb5.c -
o build/temp.macosx-11.0-x86_64-3.7/gssapi/raw/mech_krb5.o -framework GSS -DOSX_HAS_GSS_FRAMEWORK
clang: warning: -framework GSS: 'linker' input unused [-Wunused-command-line-argument]
clang -bundle -undefined dynamic_lookup -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/User
s/simlu/.asdf/installs/python/3.7.9/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/User
s/simlu/.asdf/installs/python/3.7.9/lib build/temp.macosx-11.0-x86_64-3.7/gssapi/raw/mech_krb5.o -o build/lib.macosx-11.0-x86_64-3.7/gssapi/raw/mech_krb5.cpython-37m-darwin.so -framework GSS
$

[simmel]

Ok brace yourselves because this is going to be a bumpy ride 👍

I don't feel bad anymore that I couldn't solve it ; ) Great work @jborean93 !

[samdoran]

@jborean93 You win the Internet.

[frozencemetery]

Thanks @jborean93! While python/cpython#22855 is going to be backported some, current feeling seems to be that it won't land in 3.6/3.7. So we'll need to merge your workaround and maybe we can remove it again in a few years.

[jborean93]

Except that it doesn't. Or at least some other problem arises:

@simmel that errror is because you don’t have cython installed and it’s expecting the .c files to already exist. Cython must be there at build time to generate the .c files. Normally they aren’t needed with the normal sdist but installing them from source does.

[simmel]

Right! Thank you!

…

On Thu, 2020-12-10 at 10:41:13 -0800, Jordan Borean wrote: > Except that it doesn't. Or at least some other problem arises: @simmel that errror is because you don’t have cython installed and it’s expecting the .c files to already exist. Cython must be there at build time to generate the .c files. Normally they aren’t needed with the normal sdist but installing them from source does. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #235 (comment)

[HuaiJiang]

Sorry to comment on this closed issue, but I noticed below msg , is it expected?

Skipping the cred_store extension because it is not supported by your GSSAPI implementation...