Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incompatible cached wheels may continue to be used after upgrade #3599

Open
wimglenn opened this issue May 15, 2024 · 12 comments
Open

Incompatible cached wheels may continue to be used after upgrade #3599

wimglenn opened this issue May 15, 2024 · 12 comments
Assignees
@charliermarsh
Labels
needs-mre Needs more information for reproduction

Comments

@wimglenn
Copy link
Contributor

I saw an issue installing pydantic on macOS where it uv would use an incompatible wheel for pydantic_core (with cp312-cp312-macosx_10_12_x86_64 tag) resulting in this error message:

$ .venv/bin/python3 -c 'import pydantic_core'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../.venv/lib/python3.12/site-packages/pydantic_core/__init__.py", line 6, in <module>
    from ._pydantic_core import (
ImportError: dlopen(/.../.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so, 0x0002): tried: '/.../.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/.../.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (no such file), '/.../.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))

pip would select a different one (with cp312-cp312-macosx_11_0_arm64 tag) which worked.

I had recently upgraded to uv 0.1.44 (d417daad7 2024-05-14), and a uv cache clean fixed the issue - on reinstall it used a compatible wheel.

Are the UV caches keyed off the uv version at all? Perhaps the uv installation should self-invalidate caches.
@charliermarsh
Copy link
Member

Unfortunately it's hard to help here without more information and a reliable reproduction. We intentionally don't key the cache on uv version -- instead, each of the buckets has its own version that we bump whenever the schema changes.

@charliermarsh
Copy link
Member

Do you have any idea how that wheel ended up in the cache in the first place? Either way we shouldn't be re-using it unless it appears to be compatible with the current environment. Any reproduction welcome! Otherwise I may have to close.

@zanieb zanieb added the needs-mre Needs more information for reproduction label May 15, 2024
@wimglenn
Copy link
Contributor Author

I think I can reproduce it by creating an alternate install using brew - do these installations share files at all?

$ ~/.cargo/bin/uv --version
uv 0.1.44 (d417daad7 2024-05-14)
$ arch -x86_64 brew install -q uv
==> Fetching dependencies for uv: libssh2 and libgit2
==> Fetching libssh2
==> Fetching libgit2
==> Fetching uv
==> Installing dependencies for uv: libssh2 and libgit2
==> Installing uv dependency: libssh2
🍺  /usr/local/Cellar/libssh2/1.11.0_1: 198 files, 1.1MB
==> Installing uv dependency: libgit2
🍺  /usr/local/Cellar/libgit2/1.7.2: 105 files, 4.5MB
==> Installing uv
==> Caveats
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/uv/0.1.44: 13 files, 28.4MB
==> Running `brew cleanup uv`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Caveats
==> uv
zsh completions have been installed to:
  /usr/local/share/zsh/site-functions
$ ~/.cargo/bin/uv --version
uv 0.1.44 (d417daad7 2024-05-14)
$ /usr/local/Cellar/uv/0.1.44/bin/uv --version                                  
uv 0.1.44
$ ~/.cargo/bin/uv venv     
Using Python 3.12.3 interpreter at: /usr/local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
$ /usr/local/Cellar/uv/0.1.44/bin/uv pip install pydantic-core --force-reinstall
Resolved 2 packages in 315ms
Downloaded 2 packages in 211ms
Installed 2 packages in 10ms
 + pydantic-core==2.18.2
 + typing-extensions==4.11.0
$ tail -1 .venv/lib/python3.12/site-packages/pydantic_core-2.18.2.dist-info/WHEEL
Tag: cp312-cp312-macosx_10_12_x86_64
$ .venv/bin/python -c 'import pydantic_core; print(pydantic_core.__file__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/__init__.py", line 6, in <module>
    from ._pydantic_core import (
ImportError: dlopen(/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so, 0x0002): tried: '/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (no such file), '/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/_pydantic_core.cpython-312-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))
$ ~/.cargo/bin/uv pip install pydantic-core --force-reinstall                               
Resolved 2 packages in 8ms
Installed 2 packages in 1ms
 - pydantic-core==2.18.2
 + pydantic-core==2.18.2
 - typing-extensions==4.11.0
 + typing-extensions==4.11.0
$ tail -1 .venv/lib/python3.12/site-packages/pydantic_core-2.18.2.dist-info/WHEEL
Tag: cp312-cp312-macosx_10_12_x86_64
$ ~/.cargo/bin/uv cache clean && ~/.cargo/bin/uv pip install pydantic-core --force-reinstall
Clearing cache at: /Users/wim/Library/Caches/uv
Removed 28 files (10.5MiB)
Resolved 2 packages in 245ms
Downloaded 2 packages in 277ms
Installed 2 packages in 3ms
 - pydantic-core==2.18.2
 + pydantic-core==2.18.2
 - typing-extensions==4.11.0
 + typing-extensions==4.11.0
$ tail -1 .venv/lib/python3.12/site-packages/pydantic_core-2.18.2.dist-info/WHEEL
Tag: cp312-cp312-macosx_11_0_arm64
$ .venv/bin/python -c 'import pydantic_core; print(pydantic_core.__file__)'
/private/tmp/u/.venv/lib/python3.12/site-packages/pydantic_core/__init__.py

@charliermarsh
Copy link
Member

Think I need to look at this with fresh eyes in the morning... I might be interested to see the output of the first /usr/local/Cellar/uv/0.1.44/bin/uv pip install pydantic-core --force-reinstall with the --verbose flag though. I'm not really sure why the cp312-cp312-macosx_10_12_x86_64 is being considered as compatible.

@charliermarsh
Copy link
Member

The core problem AFAICT is that for some reason cp312-cp312-macosx_10_12_x86_64 is being considered compatible with your system, which is presumedly an ARM mac?

@charliermarsh
Copy link
Member

It would be one thing if you created the venv with /usr/local/Cellar/uv/0.1.44/bin/uv, which seems to be an x86 Python, but you're creating the venv with ~/.cargo/bin/uv right?

@wimglenn
Copy link
Contributor Author

wimglenn commented May 15, 2024 •
edited
Loading

Yeah, it's a MacBook Air (Apple M1).
Right. Here's some more verbosity:

$ rm -rf .venv
$ ~/.cargo/bin/uv venv
Using Python 3.12.3 interpreter at: /usr/local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
$ ~/.cargo/bin/uv cache clean
Clearing cache at: /Users/wim/Library/Caches/uv
Removed 29 files (10.5MiB)
$ /usr/local/Cellar/uv/0.1.44/bin/uv pip install pydantic-core --force-reinstall -vv
 uv_requirements::specification::from_source source=pydantic-core
    0.002866s DEBUG uv_interpreter::virtualenv Found a virtualenv named .venv at: /private/tmp/u/.venv
    0.003150s DEBUG uv_interpreter::interpreter Probing interpreter info for: /private/tmp/u/.venv/bin/python
    0.067560s DEBUG uv_interpreter::interpreter Found Python 3.12.3 for: /private/tmp/u/.venv/bin/python
    0.068221s DEBUG uv::commands::pip::install Using Python 3.12.3 environment at .venv/bin/python
    0.068311s DEBUG uv_fs Trying to lock if free: .venv/.lock
 uv_client::linehaul::linehaul 
    0.070875s DEBUG uv_client::base_client Using registry request timeout of 30s
 uv_resolver::flat_index::from_entries 
 uv_resolver::resolver::solve 
      0.073008s   0ms DEBUG uv_resolver::resolver Solving with target Python version 3.12.3
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
        0.073901s   0ms DEBUG uv_resolver::resolver Adding direct dependency: pydantic-core*
   uv_resolver::resolver::choose_version package=pydantic-core
     uv_resolver::resolver::package_wait package_name=pydantic-core
 uv_resolver::resolver::process_request request=Versions pydantic-core
   uv_client::registry_client::simple_api package=pydantic-core
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/simple-v7/pypi/pydantic-core.rkyv
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/simple-v7/pypi/pydantic-core.rkyv"
 uv_resolver::resolver::process_request request=Prefetch pydantic-core *
          0.076620s   0ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/pydantic-core/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/pydantic-core/"
       uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/simple-v7/pypi/pydantic-core.rkyv
       uv_client::registry_client::parse_simple_api package=pydantic-core
   uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pydantic-core==2.18.2
     uv_client::registry_client::wheel_metadata built_dist=pydantic-core==2.18.2
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.msgpack
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.msgpack"
        0.269505s 194ms DEBUG uv_resolver::resolver Searching for a compatible version of pydantic-core (*)
        0.269612s 194ms DEBUG uv_resolver::resolver Selecting: pydantic-core==2.18.2 (pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl)
   uv_resolver::resolver::get_dependencies package=pydantic-core, version=2.18.2
     uv_resolver::resolver::distributions_wait version_id=pydantic-core-2.18.2
              0.270407s   1ms DEBUG uv_client::cached_client No cache entry for: https://files.pythonhosted.org/packages/15/b1/e6edfe46402a5b415fc3de86aa64fb10009b323907f8d513175bfb839aa9/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl.metadata
           uv_client::cached_client::fresh_request url="https://files.pythonhosted.org/packages/15/b1/e6edfe46402a5b415fc3de86aa64fb10009b323907f8d513175bfb839aa9/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl.metadata"
           uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.msgpack
           uv_client::registry_client::parse_metadata21 
        0.341417s  71ms DEBUG uv_resolver::resolver Adding transitive dependency for pydantic-core==2.18.2: typing-extensions>=4.6.0, <4.7.0 | >4.7.0
   uv_resolver::resolver::choose_version package=typing-extensions
     uv_resolver::resolver::package_wait package_name=typing-extensions
 uv_resolver::resolver::process_request request=Versions typing-extensions
   uv_client::registry_client::simple_api package=typing-extensions
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/simple-v7/pypi/typing-extensions.rkyv
 uv_resolver::resolver::process_request request=Prefetch typing-extensions >=4.6.0, <4.7.0 | >4.7.0
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/simple-v7/pypi/typing-extensions.rkyv"
          0.341832s   0ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/typing-extensions/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/typing-extensions/"
       uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/simple-v7/pypi/typing-extensions.rkyv
       uv_client::registry_client::parse_simple_api package=typing-extensions
   uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=typing-extensions==4.11.0
     uv_client::registry_client::wheel_metadata built_dist=typing-extensions==4.11.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.msgpack
        0.361175s  19ms DEBUG uv_resolver::resolver Searching for a compatible version of typing-extensions (>=4.6.0, <4.7.0 | >4.7.0)
        0.361188s  19ms DEBUG uv_resolver::resolver Selecting: typing-extensions==4.11.0 (typing_extensions-4.11.0-py3-none-any.whl)
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.msgpack"
   uv_resolver::resolver::get_dependencies package=typing-extensions, version=4.11.0
     uv_resolver::resolver::distributions_wait version_id=typing-extensions-4.11.0
              0.361284s   0ms DEBUG uv_client::cached_client No cache entry for: https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl.metadata
           uv_client::cached_client::fresh_request url="https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl.metadata"
           uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.msgpack
           uv_client::registry_client::parse_metadata21 
      0.379275s 306ms DEBUG uv_resolver::resolver::batch_prefetch Tried 3 versions: pydantic-core 1, root 1, typing-extensions 1
Resolved 2 packages in 308ms
    0.380907s DEBUG uv_installer::plan Identified uncached requirement: pydantic-core==2.18.2
    0.381434s DEBUG uv_installer::plan Identified uncached requirement: typing-extensions==4.11.0
 uv_installer::downloader::download total=2
   uv_installer::downloader::get_wheel name=pydantic-core==2.18.2, size=Some(1857389), url="https://files.pythonhosted.org/packages/15/b1/e6edfe46402a5b415fc3de86aa64fb10009b323907f8d513175bfb839aa9/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl"
     uv_distribution::distribution_database::get_or_build_wheel dist=pydantic-core==2.18.2
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.http
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.http"
   uv_installer::downloader::get_wheel name=typing-extensions==4.11.0, size=Some(34698), url="https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl"
     uv_distribution::distribution_database::get_or_build_wheel dist=typing-extensions==4.11.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.http
 uv_client::cached_client::from_path_sync path="/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.http"
              0.385101s   1ms DEBUG uv_client::cached_client No cache entry for: https://files.pythonhosted.org/packages/15/b1/e6edfe46402a5b415fc3de86aa64fb10009b323907f8d513175bfb839aa9/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl
           uv_client::cached_client::fresh_request url="https://files.pythonhosted.org/packages/15/b1/e6edfe46402a5b415fc3de86aa64fb10009b323907f8d513175bfb839aa9/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.whl"
              0.385193s   0ms DEBUG uv_client::cached_client No cache entry for: https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl
           uv_client::cached_client::fresh_request url="https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl"
           uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/pydantic-core/pydantic_core-2.18.2-cp312-cp312-macosx_10_12_x86_64.http
           uv_distribution::distribution_database::wheel wheel=pydantic-core==2.18.2
           uv_client::cached_client::new_cache file=/Users/wim/Library/Caches/uv/wheels-v1/pypi/typing-extensions/typing_extensions-4.11.0-py3-none-any.http
           uv_distribution::distribution_database::wheel wheel=typing-extensions==4.11.0
Downloaded 2 packages in 251ms
 uv_installer::installer::install num_wheels=2
Installed 2 packages in 9ms
 + pydantic-core==2.18.2
 + typing-extensions==4.11.0
$ tail -1 .venv/lib/python3.12/site-packages/pydantic_core-2.18.2.dist-info/WHEEL
Tag: cp312-cp312-macosx_10_12_x86_64

@charliermarsh charliermarsh self-assigned this May 15, 2024
@charliermarsh
Copy link
Member

I haven't been able to reproduce this yet. I installed an x86 Python and repeated the process above. I agree that something strange is happening above. In /usr/local/bin/python3, can you run: import platform; platform.mac_ver()?

@wimglenn
Copy link
Contributor Author 

$ /usr/local/bin/python3 -c 'import platform; print(platform.mac_ver())'
('14.4.1', ('', '', ''), 'arm64')

@wimglenn
Copy link
Contributor Author

I've updated macOS and it shows ('14.5', ('', '', ''), 'arm64') now.
I still see this issue - today it was uv pip selecting black-24.4.2-cp312-cp312-macosx_10_9_x86_64.whl when py pip collected black-24.4.2-cp312-cp312-macosx_11_0_arm64.whl.

@charliermarsh
Copy link
Member

Unfortunately I was never able to reproduce this even after installing both an x86 and an ARM Python side-by-side on my machine.

@erensahins
Copy link

erensahins commented Jun 27, 2024 •
edited
Loading

I am experiencing the same error regarding installing pydantic on MacOS M1 chip:

uv pip install pydantic==2.7.4

ImportError: dlopen(.../.venv/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-darwin.so, 0x0002): tried: '../.venv/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), 
'/System/Volumes/Preboot/Cryptexes/OS/Users/<username>/<projectname>/.venv/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-darwin.so' (no such file), '.../.venv/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

However, pip directly selects the correct architecture and installs properly.

my mac platform version:

('13.5.1', ('', '', ''), 'arm64')

Uninstalling pydantic and cleaning uv cache does not resolve the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charliermarsh charliermarsh

Labels
needs-mre Needs more information for reproduction
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@charliermarsh @zanieb @wimglenn @erensahins