Initial support for PySide6 and Qt bootstrap (#2918)

* Initial support for PySide6

- Add a new bootstrap for Qt
  - This bootstrap will be used by `pyside6-android-deploy` tool shipped
    with PySide6, which interally calls pythonforandroid using buildozer.
  - The Qt bootstrap depends on recipes PySide6 and shiboken6 among
    other mandatory recipes. The recipes for PySide6 and shiboken6
    resides in the PySide repository -
    https://code.qt.io/cgit/pyside/pyside-setup.git/tree/sources/pyside-tools/deploy_lib/android/recipes
  - The PythonActivity entrypoint class is derived from QtActivity class
    which is the main acitivty class when a Qt C++ application is
    packaged for Android. The jar containing QtActivity class is supplied
    through buildozer `android.add_jars` option.
  - The C wrapper binary to the application main.py is named as
    `main_{abi_name}` instead of just `main` for other bootstraps.
  - Multi architecture deployment is not supported at the moment.
- Adapt tests based on the new Qt bootstrap

* Add Qt boostrap to docs

- update the docs to include sections depicting the Qt boostrap.

* Tweak gradle build properties for Qt bootstrap

- Sometimes a flaky Java heap out of memory error is throw. By,
  tweaking the memory setting we can get rid of that error.

* Fix bug - check for main.py

- Qt boostrap removed from comparison in the changed line because
  its expects a value is args.launcher, which is not applicable for
  Qt boostrap. Hence, it exits with an value not found exception.
- Removing Qt boostrap from the comparison leads to checking
  for main.py or --private, which is to be done for the Qt boostrap.

* Make --init-classes truly optional

- check if empty, otherwise store empty string

* Add a non-gui test app build to CI that uses Qt bootstrap

- for the purpose of testing, the pyside6 and shiboken6 wheels, the
  extra .jar files needed and the recipes for pyside6 and shiboken6
  are manually added into testapps/on_device_unit_tests/test_qt.
  These files are normally generated by the `pyside6-android-deploy`
  tool that is shipped with PySide.
  Generating the wheels and the .jar files belongs to the scope of
  PySide and not python-for-android. Hence, they are not done here.
  This also reduces the load on the CI which will otherwise have to
  cross-compile CPython and PySide.
- The Android aarch64 wheels for testing are downloaded from Qt servers.
  These wheels are for testing purposes only and the download link will
  be updated when official PySide6 Android wheels are generated.
- Tests were added in test_requirements.py so that when running the apk
  the current date and time are printed on the terminal. The tests also
  checks shiboken6 and PySide6 module imports.

* Remove superfluous whitespace removal

- This was introduced by a VSCode setting and is unrealated to this
  patch. Although this might be good, this has to be introduced through
  a different patch.
- pyside6 recipe typo adapted in buildoptions.rst

* Add aab generation to test app with Qt bootstrap

* Fix typo in doc/source/buildoptions.rst

Co-authored-by: Mirko Galimberti <me@mirkogalimberti.com>

---------

Co-authored-by: Shyamnath Premnadh <Shyamnath.Premnadh@qt.io>
Co-authored-by: Mirko Galimberti <me@mirkogalimberti.com>

Author: 3 people

.github/workflows/push.yml

@@ -70,6 +70,8 @@ jobs:
             target: testapps-webview
           - name: service_library
             target: testapps-service_library-aar
+          - name: qt
+            target: testapps-qt
     steps:
     - name: Checkout python-for-android
       uses: actions/checkout@v4

Makefile

@@ -82,6 +82,26 @@ testapps-service_library-aar: virtualenv
     --requirements python3 \
     --arch=arm64-v8a --arch=x86 --release
 
+testapps-qt: testapps-qt/debug/apk testapps-qt/release/aab
+
+# testapps-webview/MODE/ARTIFACT
+testapps-qt/%: virtualenv
+	$(eval MODE := $(word 2, $(subst /, ,$@)))
+	$(eval ARTIFACT := $(word 3, $(subst /, ,$@)))
+	@echo Building testapps-qt for $(MODE) mode and $(ARTIFACT) artifact
+	. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \
+    python setup.py $(ARTIFACT) --$(MODE) --sdk-dir $(ANDROID_SDK_HOME) --ndk-dir $(ANDROID_NDK_HOME) \
+    --bootstrap qt \
+    --requirements python3,shiboken6,pyside6 \
+    --arch=arm64-v8a \
+	--local-recipes ./test_qt/recipes \
+	--qt-libs Core \
+	--load-local-libs plugins_platforms_qtforandroid \
+	--add-jar ./test_qt/jar/PySide6/jar/Qt6Android.jar \
+	--add-jar ./test_qt/jar/PySide6/jar/Qt6AndroidBindings.jar \
+	--permission android.permission.WRITE_EXTERNAL_STORAGE \
+	--permission android.permission.INTERNET
+
 testapps/%: virtualenv
 	$(eval $@_APP_ARCH := $(shell basename $*))
 	. $(ACTIVATE) && cd testapps/on_device_unit_tests/ && \

doc/source/buildoptions.rst

@@ -204,6 +204,49 @@ systems and frameworks.
   include multiple jar files, pass this argument multiple times.
 - ``add-source``: Add a source directory to the app's Java code.
 
+Qt
+~~
+
+This bootstrap can be used with ``--bootstrap=qt`` or by including the ``PySide6`` or
+``shiboken6`` recipe, e.g. ``--requirements=pyside6,shiboken6``. Currently, the only way
+to use this bootstrap is through `pyside6-android-deploy <https://www.qt.io/blog/taking-qt-for-python-to-android>`__
+tool shipped with ``PySide6``, as the recipes for ``PySide6`` and ``shiboken6`` are created
+dynamically. The tool builds ``PySide6`` and ``shiboken6`` wheels for a specific Android platform
+and the recipes simply unpack the built wheels. You can see the recipes `here <https://code.qt.io/cgit/pyside/pyside-setup.git/tree/sources/pyside-tools/deploy_lib/android/recipes>`__.
+
+.. note::
+  The ``pyside6-android-deploy`` tool and hence the Qt bootstrap does not support multi-architecture
+  builds currently.
+
+What are Qt and PySide?
+%%%%%%%%%%%%%%%%%%%%%%%%
+
+`Qt <https://www.qt.io/>`__ is a popularly used cross-platform C++ framework for developing
+GUI applications. `PySide6 <https://doc.qt.io/qtforpython-6/quickstart.html>`__ refers to the
+Python bindings for Qt6, and enables the Python developers access to the Qt6 API.
+`Shiboken6 <https://doc.qt.io/qtforpython-6/shiboken6/index.html>`__ is the binding generator
+tool used for generating the Python bindings from C++ code.
+
+.. note:: The `shiboken6` recipe is for the `Shiboken Python module <https://doc.qt.io/qtforpython-6/shiboken6/shibokenmodule.html>`__
+  which includes a couple of utility functions for inspecting and debugging PySide6 code.
+
+Build Options
+%%%%%%%%%%%%%
+
+``pyside6-android-deploy`` works by generating a ``buildozer.spec`` file and thereby using
+`buildozer <https://buildozer.readthedocs.io/en/latest/>`__ to control the build options used by
+``python-for-android`` with the Qt bootstrap. Apart from the general build options that works
+across all the other bootstraps, the Qt bootstrap introduces the following 3 new build options.
+
+- ``--qt-libs``: list of Qt libraries(modules) to be loaded.
+- ``--load-local-libs``: list of Qt plugin libraries to be loaded.
+- ``--init-classes``: list of Java class names to the loaded from the Qt jar files supplied through
+  the ``--add-jar`` option.
+
+These build options are automatically populated by the ``pyside6-android-deploy`` tool, but can be
+modified by updating the ``buildozer.spec`` file. Apart from the above 3 build options, the tool
+also automatically identifies the values to be fed into the cli options ``--permission``, ``--add-jar``
+depending on the PySide6 modules used by the applicaiton.
 
 Requirements blacklist (APK size optimization)
 ----------------------------------------------

doc/source/commands.rst

@@ -26,15 +26,15 @@ behaviour, though not all commands make use of them.
 
 ``--debug``
   Print extra debug information about the build, including all compilation output.
-  
+
 ``--sdk_dir``
   The filepath where the Android SDK is installed. This can
   alternatively be set in several other ways.
 
 ``--android_api``
   The Android API level to target; python-for-android will check if
   the platform tools for this level are installed.
-  
+
 ``--ndk_dir``
   The filepath where the Android NDK is installed. This can
   alternatively be set in several other ways.
@@ -74,12 +74,12 @@ supply those that you need.
   The architecture to build for. You can specify multiple architectures to build for
   at the same time. As an example ``p4a ... --arch arm64-v8a --arch armeabi-v7a ...``
   will build a distribution for both ``arm64-v8a`` and ``armeabi-v7a``.
-  
+
 ``--bootstrap BOOTSTRAP``
   The Java bootstrap to use for your application. You mostly don't
   need to worry about this or set it manually, as an appropriate
   bootstrap will be chosen from your ``--requirements``. Current
-  choices are ``sdl2`` (used with Kivy and most other apps) or ``webview``.
+  choices are ``sdl2`` (used with Kivy and most other apps), ``webview`` or ``qt``.
 
 
 .. note:: These options are preliminary. Others will include toggles

doc/source/quickstart.rst

@@ -26,8 +26,8 @@ Concepts
 
 - **bootstrap:** A bootstrap is the app backend that will start your
   application. The default for graphical applications is SDL2.
-  You can also use e.g. the webview for web apps, or service_only/service_library  for
-  background services. Different bootstraps have different additional
+  You can also use e.g. the webview for web apps, or service_only/service_library for
+  background services, or qt for PySide6 apps. Different bootstraps have different additional
   build options.
 
 *Advanced:*
@@ -281,7 +281,7 @@ Recipe management
 You can see the list of the available recipes with::
 
     p4a recipes
-    
+
 If you are contributing to p4a and want to test a recipes again,
 you need to clean the build and rebuild your distribution::
 
@@ -295,7 +295,6 @@ it (edit the ``__init__.py``)::
 
     mkdir -p p4a-recipes/myrecipe
     touch p4a-recipes/myrecipe/__init__.py
-    
 
 Distribution management
 ~~~~~~~~~~~~~~~~~~~~~~~

pythonforandroid/bootstraps/common/build/build.py

@@ -83,7 +83,7 @@ def get_bootstrap_name():
 if PYTHON is not None and not exists(PYTHON):
     PYTHON = None
 
-if _bootstrap_name in ('sdl2', 'webview', 'service_only'):
+if _bootstrap_name in ('sdl2', 'webview', 'service_only', 'qt'):
     WHITELIST_PATTERNS.append('pyconfig.h')
 
 environment = jinja2.Environment(loader=jinja2.FileSystemLoader(
@@ -543,6 +543,7 @@ def make_package(args):
     }
     if get_bootstrap_name() == "sdl2":
         render_args["url_scheme"] = url_scheme
+
     render(
         'AndroidManifest.tmpl.xml',
         manifest_path,
@@ -571,7 +572,8 @@ def make_package(args):
     render(
         'gradle.tmpl.properties',
         'gradle.properties',
-        args=args)
+        args=args,
+        bootstrap_name=get_bootstrap_name())
 
     # ant build templates
     render(
@@ -601,6 +603,26 @@ def make_package(args):
         join(res_dir, 'values/strings.xml'),
         **render_args)
 
+    # Library resources from Qt
+    # These are referred by QtLoader.java in Qt6AndroidBindings.jar
+    # qt_libs and load_local_libs are loaded at App startup
+    if get_bootstrap_name() == "qt":
+        qt_libs = args.qt_libs.split(",")
+        load_local_libs = args.load_local_libs.split(",")
+        init_classes = args.init_classes
+        if init_classes:
+            init_classes = init_classes.split(",")
+            init_classes = ":".join(init_classes)
+        arch = get_dist_info_for("archs")[0]
+        render(
+            'libs.tmpl.xml',
+            join(res_dir, 'values/libs.xml'),
+            qt_libs=qt_libs,
+            load_local_libs=load_local_libs,
+            init_classes=init_classes,
+            arch=arch
+        )
+
     if exists(join("templates", "custom_rules.tmpl.xml")):
         render(
             'custom_rules.tmpl.xml',
@@ -951,6 +973,14 @@ def create_argument_parser():
                     help='Use that parameter if you need to implement your own PythonServive Java class')
     ap.add_argument('--activity-class-name', dest='activity_class_name', default=DEFAULT_PYTHON_ACTIVITY_JAVA_CLASS,
                     help='The full java class name of the main activity')
+    if get_bootstrap_name() == "qt":
+        ap.add_argument('--qt-libs', dest='qt_libs', required=True,
+                        help='comma separated list of Qt libraries to be loaded')
+        ap.add_argument('--load-local-libs', dest='load_local_libs', required=True,
+                        help='comma separated list of Qt plugin libraries to be loaded')
+        ap.add_argument('--init-classes', dest='init_classes', default='',
+                        help='comma separated list of java class names to be loaded from the Qt jar files, '
+                             'specified through add_jar cli option')
 
     return ap
 

pythonforandroid/bootstraps/common/build/templates/gradle.tmpl.properties

@@ -1,3 +1,8 @@
+{% if bootstrap_name == "qt" %}
+# For tweaking memory settings. Otherwise, a p4a session with Qt bootstrap and PySide6 recipe
+# terminates with a Java out of memory exception
+org.gradle.jvmargs=-Xmx2500m -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
+{% endif %}
 {% if args.enable_androidx %}
 android.useAndroidX=true
 android.enableJetifier=true

pythonforandroid/bootstraps/qt/__init__.py

@@ -0,0 +1,53 @@
+import sh
+from os.path import join
+from pythonforandroid.toolchain import (
+    Bootstrap, current_directory, info, info_main, shprint)
+from pythonforandroid.util import ensure_dir, rmdir
+
+
+class QtBootstrap(Bootstrap):
+    name = 'qt'
+    recipe_depends = ['python3', 'genericndkbuild', 'PySide6', 'shiboken6']
+    # this is needed because the recipes PySide6 and shiboken6 resides in the PySide Qt repository
+    # - https://code.qt.io/cgit/pyside/pyside-setup.git/
+    # Without this some tests will error because it cannot find the recipes within pythonforandroid
+    # repository
+    can_be_chosen_automatically = False
+
+    def assemble_distribution(self):
+        info_main("# Creating Android project using Qt bootstrap")
+
+        rmdir(self.dist_dir)
+        info("Copying gradle build")
+        shprint(sh.cp, '-r', self.build_dir, self.dist_dir)
+
+        with current_directory(self.dist_dir):
+            with open('local.properties', 'w') as fileh:
+                fileh.write('sdk.dir={}'.format(self.ctx.sdk_dir))
+
+        arch = self.ctx.archs[0]
+        if len(self.ctx.archs) > 1:
+            raise ValueError("Trying to build for more than one arch. Qt bootstrap cannot handle that yet")
+
+        info(f"Bootstrap running with arch {arch}")
+
+        with current_directory(self.dist_dir):
+            info("Copying Python distribution")
+
+            self.distribute_libs(arch, [self.ctx.get_libs_dir(arch.arch)])
+            self.distribute_aars(arch)
+            self.distribute_javaclasses(self.ctx.javaclass_dir,
+                                        dest_dir=join("src", "main", "java"))
+
+            python_bundle_dir = join(f'_python_bundle__{arch.arch}', '_python_bundle')
+            ensure_dir(python_bundle_dir)
+            site_packages_dir = self.ctx.python_recipe.create_python_bundle(
+                join(self.dist_dir, python_bundle_dir), arch)
+
+        if not self.ctx.with_debug_symbols:
+            self.strip_libraries(arch)
+        self.fry_eggs(site_packages_dir)
+        super().assemble_distribution()
+
+
+bootstrap = QtBootstrap()

pythonforandroid/bootstraps/qt/build/.gitignore

@@ -0,0 +1,14 @@
+.gradle
+/build/
+
+# Ignore Gradle GUI config
+gradle-app.setting
+
+# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)
+!gradle-wrapper.jar
+
+# Cache of project
+.gradletasknamecache
+
+# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898
+# gradle/wrapper/gradle-wrapper.properties

pythonforandroid/bootstraps/qt/build/blacklist.txt

@@ -0,0 +1,70 @@
+# prevent user to include invalid extensions
+*.apk
+*.aab
+*.apks
+*.pxd
+
+# eggs
+*.egg-info
+
+# unit test
+unittest/*
+
+# python config
+config/makesetup
+
+# unused encodings
+lib-dynload/*codec*
+encodings/cp*.pyo
+encodings/tis*
+encodings/shift*
+encodings/bz2*
+encodings/iso*
+encodings/undefined*
+encodings/johab*
+encodings/p*
+encodings/m*
+encodings/euc*
+encodings/k*
+encodings/unicode_internal*
+encodings/quo*
+encodings/gb*
+encodings/big5*
+encodings/hp*
+encodings/hz*
+
+# unused python modules
+bsddb/*
+wsgiref/*
+hotshot/*
+pydoc_data/*
+tty.pyo
+anydbm.pyo
+nturl2path.pyo
+LICENCE.txt
+macurl2path.pyo
+dummy_threading.pyo
+audiodev.pyo
+antigravity.pyo
+dumbdbm.pyo
+sndhdr.pyo
+__phello__.foo.pyo
+sunaudio.pyo
+os2emxpath.pyo
+multiprocessing/dummy*
+
+# unused binaries python modules
+lib-dynload/termios.so
+lib-dynload/_lsprof.so
+lib-dynload/*audioop.so
+lib-dynload/_hotshot.so
+lib-dynload/_heapq.so
+lib-dynload/_json.so
+lib-dynload/grp.so
+lib-dynload/resource.so
+lib-dynload/pyexpat.so
+lib-dynload/_ctypes_test.so
+lib-dynload/_testcapi.so
+
+# odd files
+plat-linux3/regen

pythonforandroid/bootstraps/qt/build/jni/Application.mk

@@ -0,0 +1,8 @@
+
+# Uncomment this if you're using STL in your project
+# See CPLUSPLUS-SUPPORT.html in the NDK documentation for more information
+# APP_STL := stlport_static
+
+# APP_ABI := armeabi armeabi-v7a x86
+APP_ABI := $(ARCH)
+APP_PLATFORM := $(NDK_API)

pythonforandroid/bootstraps/qt/build/jni/application/src/Android.mk

@@ -0,0 +1,18 @@
+LOCAL_PATH := $(call my-dir)
+
+include $(CLEAR_VARS)
+
+LOCAL_MODULE := main_$(PREFERRED_ABI)
+
+# Add your application source files here...
+LOCAL_SRC_FILES := start.c
+
+LOCAL_CFLAGS += -I$(PYTHON_INCLUDE_ROOT) $(EXTRA_CFLAGS)
+
+LOCAL_SHARED_LIBRARIES := python_shared
+
+LOCAL_LDLIBS := -llog $(EXTRA_LDLIBS)
+
+LOCAL_LDFLAGS += -L$(PYTHON_LINK_ROOT) $(APPLICATION_ADDITIONAL_LDFLAGS)
+
+include $(BUILD_SHARED_LIBRARY)