Supporting Python 3.13

[junkmd]

Python 3.13 is scheduled for release in October.
https://devguide.python.org/versions/#supported-versions

A version that explicitly supports Python 3.13 will likely be released after GitHub Actions begins supporting the stable version of Python 3.13, to ensure compatibility in the CI pipeline.

We may upgrade the version of black, or perhaps take this opportunity to adopt another formatter like ruff.

Feel free to share your thoughts with the community.

[nehaljwani]

A test build for comtypes for Python 3.13 succeeds, but the test fails:

===== testing package: comtypes-1.4.7-py313hfa70ccb_1 =====
running run_test.py
Traceback (most recent call last):
  File "D:\bld\comtypes_1727009486116\test_tmp\run_test.py", line 5, in <module>
    from comtypes.client import CreateObject, CoGetObject, GetActiveObject
  File "D:\bld\comtypes_1727009486116\_test_env\Lib\site-packages\comtypes\__init__.py", line 177, in <module>
    from comtypes._post_coinit import _shutdown
  File "D:\bld\comtypes_1727009486116\_test_env\Lib\site-packages\comtypes\_post_coinit\__init__.py", line 15, in <module>
    from comtypes._post_coinit.unknwn import _shutdown  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\bld\comtypes_1727009486116\_test_env\Lib\site-packages\comtypes\_post_coinit\unknwn.py", line 371, in <module>
    class _compointer_base(c_void_p, metaclass=_compointer_meta):
    ...<89 lines>...
            return value.QueryInterface(cls.__com_interface__)
  File "D:\bld\comtypes_1727009486116\_test_env\Lib\site-packages\comtypes\_post_coinit\unknwn.py", line 93, in __new__
    p = type(_compointer_base)(
             ^^^^^^^^^^^^^^^^
NameError: name '_compointer_base' is not defined. Did you mean: '_compointer_meta'?

[junkmd]

Hi, @nehaljwani

Surprisingly, it seems that Python 3.13 (at least in the pre-release rc2) has introduced a breaking change (or a regression) in the resolution order for metaclass handling.
comtypes implements a complex metaclass to make the IUnknown interface behave like ctypes-classes.
For instance, _compointer_meta is the "magic" used to prevent metaclass conflicts.

I am not familiar with the conda build process, nor do I know if other projects have encountered similar issues with metaclasses in combination with Python 3.13.
If you are aware of anything regarding these, I would appreciate your insights.

Also, the What’s New In Python 3.13 document does not mention anything about changes related to metaclass behavior.
This might be a regression in CPython, so it could be worth reporting it as an issue with CPython.

[junkmd]

I made some modifications to the comtypes codebase to see if I could address this issue, but it doesn't seem to be that simple.

Even after changing the definition order of _compointer_base in _post_coinit/unknwn.py, a NameError still occurred.
When I moved it to a separate module and imported it within the _cominterface_meta.__new__ method, an ImportError occurred.

This might be a similar case to #212, where a bug fix on the CPython side is needed.

[junkmd]

I found pyglet/pyglet#1196 and pyglet/pyglet#1199.

Like comtypes, the pyglet package relies on ctypes to define COM interfaces and COM methods.

As mentioned in pyglet/pyglet#1196 (comment), I suspect that the error in comtypes is also caused by python/cpython#114314.

Following the solution implemented in pyglet, I tried defining __init__ in addition to __new__ for the metaclass.
This failed in Python versions below 3.13 but succeeded in Python 3.13.
After applying a makeshift patch (04b766a) that adds a version bridge for Python, I tested it on various Python versions.
The metaclass-related error no longer occurred, and the tests no longer crashed entirely.

I'd like to hear the community's opinion on whether we should apply this to a new release.
Additionally, the maintainers of ctypes might have ideas for handling such cases, so I'm also considering escalating this to CPython.

Aside from the metaclass issue, a new problem has emerged: tests using _midlSAFEARRAY, such as test_midl_safearray_create, are failing in Python 3.13 as shown below.

ERROR: test_midl_safearray_create (unittest.loader._FailedTest.test_midl_safearray_create)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_midl_safearray_create
Traceback (most recent call last):
  File "D:\a\comtypes\comtypes\comtypes\safearray.py", line 66, in _midlSAFEARRAY
    return POINTER(_safearray_type_cache[itemtype])  # type: ignore
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: <class 'ctypes.c_long'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.0-rc.2\x64\Lib\unittest\loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\hostedtoolcache\windows\Python\3.13.0-rc.2\x64\Lib\unittest\loader.py", line 339, in _get_module_from_name
    __import__(name)
    ~~~~~~~~~~^^^^^^
  File "D:\a\comtypes\comtypes\comtypes\test\test_midl_safearray_create.py", line 12, in <module>
    GetModule("UIAutomationCore.dll")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\comtypes\comtypes\comtypes\client\_generate.py", line 128, in GetModule
    return ModuleGenerator(tlib, pathname).generate()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "D:\a\comtypes\comtypes\comtypes\client\_generate.py", line 245, in generate
    return [_create_module(name, code) for (name, code) in codebases][-1]
            ~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "D:\a\comtypes\comtypes\comtypes\client\_generate.py", line 217, in _create_module
    return _my_import(modulename)
  File "D:\a\comtypes\comtypes\comtypes\client\_generate.py", line 28, in _my_import
    return importlib.import_module(fullname)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.13.0-rc.2\x64\Lib\importlib\__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\comtypes\comtypes\comtypes\gen\_944DE083_8FB8_45CF_BCB7_C477ACB2F897_0_1_0.py", line 711, in <module>
    (['out', 'retval'], POINTER(_midlSAFEARRAY(c_int)), 'runtimeId')
                                ~~~~~~~~~~~~~~^^^^^^^
  File "D:\a\comtypes\comtypes\comtypes\safearray.py", line 68, in _midlSAFEARRAY
    sa_type = _make_safearray_type(itemtype)
  File "D:\a\comtypes\comtypes\comtypes\safearray.py", line 111, in _make_safearray_type
    @Patch(POINTER(sa_type))
           ~~~~~~~^^^^^^^^^
TypeError: _type_ must have storage info

Since I'm not very familiar with safearrays, I'd like to ask @bennyrowland and @geppi, who have worked for safearrays in the past.

[jaraco]

+1 for ruff over black

Also, +1 to reporting this issue to CPython, especially if you can distill the issue to something that's not specific to comtypes or pyglet. It will be useful to have documented the breaking change somewhere.

[junkmd]

It appears that changes have occurred in Python 3.13 regarding the handling of constructors, such as __new__ and __init__, in ctypes classes.
Therefore, I thought that by modifying the part of _midlSAFEARRAY's internal processing that instantiates the metaclass (i.e., where the class is dynamically defined), it might work as before.

When I modified the codebase of safearray._make_safearray_type as shown below, the tests passed without failure in Python 3.13.

    meta = type(_safearray.tagSAFEARRAY)
-   sa_type = meta.__new__(
-       meta, "SAFEARRAY_%s" % itemtype.__name__, (_safearray.tagSAFEARRAY,), {}
-   )
+   sa_type = meta(f"SAFEARRAY_{itemtype.__name__}", (_safearray.tagSAFEARRAY,), {})

Unlike the changes made to _compointer_meta and _coclass_meta, this modification didn’t require a version-bridging solution, and the same codebase ran successfully from Python 3.8 to Python 3.12.

To share the differences between the "working on Python 3.13" codebase and the previous one, as well as the CI pipeline results, I created a draft PR as #623.
The PR may be rebased, so I’m also sharing with the community that the initial production code changes in that PR were 981d4b4..6f036d4.

In my next free time, I plan to escalate these results to CPython and seek the opinions of the ctypes maintainers regarding this modification (in addition to the GitHub issue, https://discuss.python.org/ might also be a good place).
My goal is to share valuable information not only with the comtypes community but also with the entire Python community regarding metaprogramming involving ctypes, and to discuss what can be done to prevent future regressions.

[junkmd]

I reported the breaking of backward compatibility with ctypes and metaclasses in python/cpython#124520.
By a ctypes maintainer, python/cpython#124546 was submitted.

I also received some advice, so I'm thinking of rewriting the version bridge as well.

[cfarrow]

Great job hunting down where this issue came from. I read through python/cpython#124520 and your assessment seems spot on. I don't understand the ctypes changes well enough to predict if anything else will break because of this. It seems that you've already checked where __new__ is being called and put in a version bridge. Beyond that, I think we trust the tests.