Android build fails if py command finds Python 3.12: ModuleNotFoundError: No module named 'pip._vendor.six.moves'

[rmartin16]

Describe the bug

Android build fails during bootstrapping pip install. This may not be the best place to log this but wanted to report and go from there.

This testing is simply using Briefcase's helloworld sample project.

> Task :app:preBuild UP-TO-DATE
> Task :app:preDebugBuild UP-TO-DATE
> Task :app:mergeDebugNativeDebugMetadata NO-SOURCE
> Task :app:extractPythonBuildPackages

> Task :app:generateDebugPythonRequirements FAILED
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradle\app\build\generated\python\bp\chaquopy\pip_install.py", line 23, in <module>
    from pip._internal.utils.misc import rmtree
  File "C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradle\app\build\generated\python\bp\pip\_internal\utils\misc.py", line 20, in <module>
    from pip._vendor import pkg_resources
  File "C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradle\app\build\generated\python\bp\pip\_vendor\pkg_resources\__init__.py", line 58, in <module>
    from pip._vendor.six.moves import urllib, map, filter
ModuleNotFoundError: No module named 'pip._vendor.six.moves'

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'py'' finished with non-zero exit value 1

Steps to reproduce

Using Python 3.8 or 3.9, run briefcase build android.

Expected behavior

Build is successful.

Screenshots

No response

Environment

• Operating System: Windows 11
• Python version: Python 3.8
• Software versions:
  • Briefcase: 0.3.16.dev282+ge1b9deee and 0.3.15

Logs

briefcase.2023_10_05-06_11_14.build_normal_debug.log
briefcase.2023_10_05-06_13_34.build_verbose_debug.log

Additional context

No response

[freakboy3742]

I can't reproduce this on macOS. With a clean Python3.8 venv, Briefcase at 0.3.16.dev275+g939b17a2, I get a clean build of a Helloworld project.

Pip is v23.2.1, Setuptools v68.2.2, wheel v0.41.2; pip freeze returns:

arrow==1.3.0
binaryornot==0.4.4
-e git+ssh://git@github.com/freakboy3742/briefcase@939b17a25d693f18f1878f9964ada1d6a36102dc#egg=briefcase
build==1.0.3
certifi==2023.7.22
chardet==5.2.0
charset-normalizer==3.3.0
click==8.1.7
cookiecutter==2.4.0
dmgbuild==1.6.1
ds-store==1.3.1
gitdb==4.0.10
GitPython==3.1.37
idna==3.4
importlib-metadata==6.8.0
importlib-resources==5.9.0
Jinja2==3.1.2
mac-alias==2.2.2
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mdurl==0.1.2
packaging==23.2
platformdirs==3.11.0
psutil==5.9.5
Pygments==2.16.1
pyproject_hooks==1.0.0
python-dateutil==2.8.2
python-slugify==8.0.1
PyYAML==6.0.1
requests==2.31.0
rich==13.6.0
six==1.16.0
smmap==5.0.1
text-unidecode==1.3
tomli==2.0.1
tomli_w==1.0.0
types-python-dateutil==2.8.19.14
typing_extensions==4.8.0
urllib3==2.0.6
zipp==3.17.0

[rmartin16]

Couldn't reproduce on Linux either. If you look at pip._vendor.six.moves, it is doing some very interesting things with imports. I think the issue might stem from the use of py....but that's just a hunch.....there's also similar-looking tickets for pypa. I may try to dig in more....there's just several layers here I'm not familiar with at all.

[mhsmith]

This appears to be a Chaquopy problem, but I've never seen it before. Chaquopy tries to run it's own bundled copy of pip in an isolated environment, so it shouldn't be affected by anything on the build machine other than Python itself. But some things have leaked through in the past, such as via environment variables (e.g. chaquo/chaquopy#777). PY_PYTHON=3.1 looks a bit suspicious here.

[mhsmith]

I take it python/bp/pip/_vendor/six.py actually exists, and hasn't been deleted by an anti-virus or something?

[rmartin16]

PY_PYTHON=3.1 looks a bit suspicious here.

Agreed; I thought I was using an environment without that. Nonetheless, the same is happening without it.

briefcase.2023_10_06-19_39_47.build.log

I take it python/bp/pip/_vendor/six.py actually exists, and hasn't been deleted by an anti-virus or something?

It does exist.

❯ Get-Item C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradle\app\build\generated\python\bp\pip\_vendor\pkg_resources\__init__.py | Format-List

    Directory: C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradle\app\build\generate
d\python\bp\pip\_vendor\pkg_resources

Name           : __init__.py
Length         : 108277
CreationTime   : 10/6/2023 7:39:45 PM
LastWriteTime  : 10/6/2023 7:39:45 PM
LastAccessTime : 10/6/2023 7:42:14 PM
Mode           : -a---
LinkType       :
Target         :
VersionInfo    : File:             C:\Users\user\github\beeware\briefcase\tmp\helloworld\build\helloworld\android\gradl
                 e\app\build\generated\python\bp\pip\_vendor\pkg_resources\__init__.py
                 InternalName:
                 OriginalFilename:
                 FileVersion:
                 FileDescription:
                 Product:
                 ProductVersion:
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:

I've been trying to get rid of the issue by recreating the virtual environment as well as the helloworld project...but it persists...

[rmartin16]

From the verbose Gradle log, I see that pip is being bootstrapped with py -3 -S -m chaquopy.pip_install. When I run py -3 from the command line, it runs Python 3.12. This is obviously a recent change so I uninstalled the Windows Store version of 3.12 and now py -3 runs Python 3.11. When I re-ran briefcase build android, it doesn't error now.

[rmartin16]

This seems to come down to relying on the py command to find versions of Python in the user's environment. The build process appears to try to find Python first specifically, then more generically; for instance, trying py -3.8 and if that fails, py -3. For me, py can't find the Python 3.8 I have installed; so, it ends up using whatever py -3 returns. Alternatively, py -3.10 does succeed because my version 3.8 was installed completely differently than my version 3.10.

Nonetheless, I would expect users to hit this if they are using Python installations that don't really interact or depend on the py command.

[rmartin16]

I guess the question for me is: should the build process really proceed forward if it's using a different version of Python than the developer is trying to use?

[mhsmith]

OK, this is the downside of using a bundled copy of pip: eventually it will become incompatible with new versions of Python. Testing with 3.12 was already on my list for the next Chaquopy version (chaquo/chaquopy#931), but I didn't get to it in time for the 3.12 stable release.

Upstreaming our patches into pip would make it easier to use a newer version of pip, and avoid this kind of thing in the future (chaquo/chaquopy#981).

I'd prefer for Chaquopy not to require a perfect Python version match, as this could raise the barrier to entry for new users. The only other reason why a Python version mismatch could matter is bytecode compilation, and Chaquopy already handles that by showing a warning and allowing the compilation to happen on the target device.

However, we could improve the way in which it searches for a matching Python version. Currently the preference order on WIndows is ['py -3.x', 'py -3', 'python'], and on Unix ['python3.x', 'python3', 'python']. The commands with version numbers will use the current virtual environment on Unix, but not on Windows.

So my suggestion is to move python up to the second position on the list, between the 3.x command and the 3 command. And in case python happens to be Python 2 or some other version that's too old, instead of testing it by passing --version as we currently do, run our existing minimum version check.

There is one complication: the Gradle daemon doesn't update its PATH, so it won't notice if you switch between virtual environments (gradle/gradle#10483). But I think this change would fix a lot of common cases without making any of them worse.

[rmartin16]

There is one complication: the Gradle daemon doesn't update its PATH, so it won't notice if you switch between virtual environments (gradle/gradle#10483). But I think this change would fix a lot of common cases without making any of them worse.

I was wondering about this as well. I've noticed if I change the Python version for Briefcase by switching virtual environments, a previous version of Python is used.

More specifically, if I first use Python 3.10 to run briefcase create android && briefcase build android and then switch Python 3.8 and run briefcase build android, the build process still uses Python 3.10. This remains true even if I terminate the Gradle daemon.

So, what's driving the persistent use of Python 3.10 here? Is it the Python version in build.gradle?

If so, I really think we need we should inform users this is happening and how it might be mitigated...if we can't prevent it altogether. Especially since a common MO right now is to tell users to use an older Python for package availability...but then using an older Python doesn't seem to work either and it's rather opaque why.

[rmartin16]

Although, thinking about this more broadly, if a user changes virtual environments after briefcase create is run, then Briefcase should probably universally inform the user that briefcase create should be run again if they intend for a different version of Python to be used for the project.

Some experiments:

• Flatpak: the build process continues using the Python used when the template was rolled out
• Linux native: a specific version of Python is enforced
• Windows: the app will bundle the version of Python used to create it...but will later install new packages based on the version of Python the user is currently using (!!!)

Seems relevant:

• Make create, update, build and run automatically execute any of the earlier steps which are out of date #807

[mhsmith]

So, what's driving the persistent use of Python 3.10 here? Is it the Python version in build.gradle?

Yes. So if there's a py -3.10 anywhere on the PATH (or python3.10 on Linux and Mac), then that will take priority over the python in your virtual environment.

Also relevant:

• Update templated content after initial call to create #472

[mhsmith]

The original error will be fixed by Chaquopy 15.0.1, which should be released within the next couple of weeks. This issue should be left open until Briefcase has updated to that.

[Pairman]

Happened on my machine (Windows 11 amd64). After I removed Python 3.12, installed 3.11, rebooted and recreated venv, I still get this error running briefcase package android:

           FAILURE: Build failed with an exception.
subprocess.py:685
           * Where:
subprocess.py:685
           Build file 'C:\Users\Pairman\bw\xdcheckin\build\xdcheckin\android\gradle\app\build.gradle' line: 25
subprocess.py:685
           * What went wrong:
subprocess.py:685
           A problem occurred evaluating project ':app'.
subprocess.py:685
           > Invalid Python version '3.12'. Available versions are [3.8, 3.9, 3.10, 3.11].

While there's only python command on my machine, no py.
briefcase.2023_12_02-10_07_34.package.log

[mhsmith]

After changing your venv's Python version, you'll need to re-run briefcase create android.