Tag macosx_10_9_universal2 not accepted when running native arm64 (on macOS 11)

[rluce]

Environment

• MacBook Pro with M1 chip
• Released Python 3.9.1 universal2 installer
• Latest pip, setuptools and wheel

Description

Running the arm64 portion of Python 3.9.1 to install a universal2 binary wheel ("macosx_10_9_universal2" is the default tag) fails. On contrary, using the x86_64 portion of the very same binary the installation succeeds. For me the expected behaviour is that such wheels can be installed with both the arm64 and x86_64 portions on macOS 11.

The offending portion that pip rejects is the 10_9 part. If I manually rename this portion of the wheel file to 11_0, the installation succeeds. Again, for me the expected behaviour is that such wheels can be installed on macOS 11

What did you want to do?

Install a universal2 binary wheel on macOS 11 using Python 3.9.1. Simplified setup using a dummy wheel:

% touch asdf-1.0.0-cp39-cp39-macosx_10_9_universal2.whl 
% python -m pip install asdf-1.0.0-cp39-cp39-macosx_10_9_universal2.whl 

Output

ERROR: asdf-1.0.0-cp39-cp39-macosx_10_9_universal2.whl is not a supported wheel on this platform.

Additional information

Using the x86_64 portion, the wheel tag is accepted (and we get the expected error from using a dummy wheel):

% arch -x86_64 python -m pip install asdf-1.0.0-cp39-cp39-macosx_10_9_universal2.whl
Processing ./asdf-1.0.0-cp39-cp39-macosx_10_9_universal2.whl
[ ... ]
zipfile.BadZipFile: File is not a zip file

Version of the interpreter, showing that the system platform matches the wheel tag:

(Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:44:01) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig; sysconfig.get_platform()
'macosx-10.9-universal2'

Version of installed packages:

pip        21.0.dev0
setuptools 51.1.1
wheel      0.36.2

[rluce]

I think PR pypa/wheel#390 is related, but I'm not sure to which extend. Maybe @ronaldoussoren knows off the top of his head what is going on here?

[uranusjr]

pypa/wheel#390 is for creating wheels, so this is not related.

Can you try what this prints for you?

from pip._vendor.packaging.tags import cpython_tags
for t in cpython_tags(): print(str(t))

You may want to report this to https://github.com/pypa/packaging instead (which pip uses underneath, and this error is emitted if pip detects a tag not recognisable by the library); you’d more easily get attention from people with relevant knowledge there.

[pradyunsg]

Could you also provide the output of pip debug --verbose on your system?

[rluce]

Can you try what this prints for you?

from pip._vendor.packaging.tags import cpython_tags
for t in cpython_tags(): print(str(t))

>>> from pip._vendor.packaging.tags import cpython_tags
>>> for t in cpython_tags(): print(str(t))
... 
cp39-cp39-macosx_11_0_arm64
cp39-cp39-macosx_11_0_universal2
cp39-abi3-macosx_11_0_arm64
cp39-abi3-macosx_11_0_universal2
cp39-none-macosx_11_0_arm64
cp39-none-macosx_11_0_universal2
cp38-abi3-macosx_11_0_arm64
cp38-abi3-macosx_11_0_universal2
cp37-abi3-macosx_11_0_arm64
cp37-abi3-macosx_11_0_universal2
cp36-abi3-macosx_11_0_arm64
cp36-abi3-macosx_11_0_universal2
cp35-abi3-macosx_11_0_arm64
cp35-abi3-macosx_11_0_universal2
cp34-abi3-macosx_11_0_arm64
cp34-abi3-macosx_11_0_universal2
cp33-abi3-macosx_11_0_arm64
cp33-abi3-macosx_11_0_universal2
cp32-abi3-macosx_11_0_arm64
cp32-abi3-macosx_11_0_universal2

Could you also provide the output of pip debug --verbose on your system?

% /usr/local/bin/pip3.9 debug --verbose
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.3.3 from /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip (python 3.9)
sys.version: 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:44:01) 
[Clang 12.0.0 (clang-1200.0.32.27)]
sys.executable: /usr/local/bin/python3.9
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: darwin
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.4
  contextlib2==0.6.0.post1 (Unable to locate actual module version, using vendor.txt specified version)
  distlib==0.3.1
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  ipaddress==1.0.23
  msgpack==1.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==20.8
  pep517==0.9.1
  progress==1.5
  pyparsing==2.4.7
  requests==2.25.0
  certifi==2020.11.08
  chardet==3.0.4
  idna==2.10
  urllib3==1.26.2
  resolvelib==0.5.3
  retrying==1.3.3 (Unable to locate actual module version, using vendor.txt specified version)
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.15.0
  toml==0.10.2
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 54
  cp39-cp39-macosx_11_0_arm64
  cp39-cp39-macosx_11_0_universal2
  cp39-abi3-macosx_11_0_arm64
  cp39-abi3-macosx_11_0_universal2
  cp39-none-macosx_11_0_arm64
  cp39-none-macosx_11_0_universal2
  cp38-abi3-macosx_11_0_arm64
  cp38-abi3-macosx_11_0_universal2
  cp37-abi3-macosx_11_0_arm64
  cp37-abi3-macosx_11_0_universal2
  cp36-abi3-macosx_11_0_arm64
  cp36-abi3-macosx_11_0_universal2
  cp35-abi3-macosx_11_0_arm64
  cp35-abi3-macosx_11_0_universal2
  cp34-abi3-macosx_11_0_arm64
  cp34-abi3-macosx_11_0_universal2
  cp33-abi3-macosx_11_0_arm64
  cp33-abi3-macosx_11_0_universal2
  cp32-abi3-macosx_11_0_arm64
  cp32-abi3-macosx_11_0_universal2
  py39-none-macosx_11_0_arm64
  py39-none-macosx_11_0_universal2
  py3-none-macosx_11_0_arm64
  py3-none-macosx_11_0_universal2
  py38-none-macosx_11_0_arm64
  py38-none-macosx_11_0_universal2
  py37-none-macosx_11_0_arm64
  py37-none-macosx_11_0_universal2
  py36-none-macosx_11_0_arm64
  py36-none-macosx_11_0_universal2
  py35-none-macosx_11_0_arm64
  py35-none-macosx_11_0_universal2
  py34-none-macosx_11_0_arm64
  py34-none-macosx_11_0_universal2
  py33-none-macosx_11_0_arm64
  py33-none-macosx_11_0_universal2
  py32-none-macosx_11_0_arm64
  py32-none-macosx_11_0_universal2
  py31-none-macosx_11_0_arm64
  py31-none-macosx_11_0_universal2
  py30-none-macosx_11_0_arm64
  py30-none-macosx_11_0_universal2
  cp39-none-any
  py39-none-any
  py3-none-any
  py38-none-any
  py37-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

I'll ping pypa/packaging then, thanks!

[rluce]

Actually if anyone with write permissions on pypa/packaging would transfer this issue, we could skip the duplicate. I'll wait an hour or so now before opening another issue over there.

[pradyunsg]

How are you getting the 10_9 tag for the universal2 wheel?

[rluce]

For the actual wheel I was looking at: I was building it through the standard toolchain python setup.py bdist_wheel, using the exact same Python interpreter on the exact same macOS 11 machine. It would take me a while to construct a minimal source example though.

[rluce]

That said, getting the 10_9 platform tag was my expectation -- I wanted a binary wheel whose x86_64 portion would be compatible down to macOS 10.9 and whose arm64 portion would run on all supported versions too (which currently only is 11.X...)

[ronaldoussoren]

How are you getting the 10_9 tag for the universal2 wheel?

Using a recentish version of wheel and the universal2 build of CPython. I don't have a link to the relevant PR at hand, but the basic idea is that a "macosx_10_9_universal2" wheel will work on all mac systems with an x86_64 or arm64 CPU running macOS 10.9 or later.

[pradyunsg]

Ah.

No one has updated packaging.tags for this change in CPython yet, so I'd request someone familiar with the situation to go ahead and file a PR for this. :)

[ronaldoussoren]

No one has updated packaging.tags for this change in CPython yet, so I'd request someone familiar with the situation to go ahead and file a PR for this. :)

I would have expected that #319 would have done the trick, but apparently not. I just received my M1 system and am currently geeking on on configuring it.

What I don't quite understand is that pip install pyobjc-core tries to build from source, while pip install --only-binary :all: pyobjc-core does use the universal2 binary wheel, but only the "macosx_11_0_universal2" wheel for PyObjC 7.0.1 and not the "macosx_10_9_universal2" one for PyObjC 7.1.

I guess its time to clone the repo and try to understand what's going on here.

[pradyunsg]

We're basically using the tags from this package as-is. Running pip debug -v will print the list of tags that pip tries to use with wheels.

[ronaldoussoren]

pip debug -v shows that pip only looks at 11.0 wheels (as was to be expected given this issue).