Migrate to Python 3.12, require it #421

I realize you support older, and may want to, I'm just thinking if it could be better to only support 3.12+ already.

What might be reasons and pros and cons? I see:

Porting to Python 3.12

• Legacy Unicode APIs based on Py_UNICODE* representation has been removed. Please migrate to APIs based on UTF-8 or wchar_t*.

They want to move to UTF-8, I believe it's not yet the default encoding, but it's been an alternative redundant encoding for a long time.

When you call to or from Python, then I believe it's non-copying, you can reuse arrays in either direction, and that would be ideal for strings too and all else. What are the exceptions? Strings are UTF-8 in Julia and need to be converted (or not?), if calling Python, though probably if Julia is called with a Python string.

Also Dicts are unordered in Julia, so I'm not sure what happens. Possibly OrderedDefaultDict could be handled better, or if were Julia's default, but right now dicts can't be shared in either direction. Are there more exceptions?

Python 3.12 should work already, so you don't have to do anything per se. And PythonCall will install a Python for you (now a smaller download since a lot of deprecated/outdated stuff dropped), but if you require 3.12 or higher it will be 3.12 for now. I think you already get the latest anyway? Unless you ask for older directly or indirectly, i.e. some package forces it. So then it couldn't be used. Going forward will not be a problem.

The only point of restricting is if you add capability that only exists in 3.12, so that you can use it. Older versions of your package would still work with older Python and that could be documented, and even supported for a while.

So what do you think, is there anything new and interesting in 3.12 you want to take advantage of?

I haven't looked over everything, but there are intriguing changes (probably largest Python release in a while):

[Python already has a (limited) stable C API, that I suppose you use. Plus I now see this new unstable.]

New Features¶

• PEP 697: Introduce the Unstable C API tier, intended for low-level tools like debuggers and JIT compilers. This API may change in each minor release of CPython without deprecation warnings. Its contents are marked by the PyUnstable_ prefix in names.

[..]

• PEP 697: Add an API for extending types whose instance memory layout is opaque:
  Add the new limited C API function PyType_FromMetaclass(), which generalizes the existing PyType_FromModuleAndSpec() using an additional metaclass argument. (Contributed by Wenzel Jakob in gh-93012.)

• API for creating objects that can be called using the vectorcall protocol was added to the Limited API:
  [..]
  The Py_TPFLAGS_HAVE_VECTORCALL flag is now removed from a class when the class’s __call__() method is reassigned. This makes vectorcall safe to use with mutable types (i.e. heap types without the immutable flag, Py_TPFLAGS_IMMUTABLETYPE). Mutable types that do not override tp_call now inherit the Py_TPFLAGS_HAVE_VECTORCALL flag. (Contributed by Petr Viktorin in gh-93274.)
  The Py_TPFLAGS_MANAGED_DICT and Py_TPFLAGS_MANAGED_WEAKREF flags have been added. This allows extensions classes to support object __dict__ and weakrefs with less bookkeeping, using less memory and with faster access.

[..]

• The C API now permits registering callbacks via PyDict_AddWatcher(), PyDict_Watch() and related APIs to be called whenever a dictionary is modified. This is intended for use by optimizing interpreters, JIT compilers, or debuggers. (Contributed by Carl Meyer in gh-91052.)

[..]

• PEP 683: Introduce Immortal Objects, which allows objects to bypass reference counts, and related changes to the C-API:

[..]

• PEP 684: Add the new Py_NewInterpreterFromConfig() function and PyInterpreterConfig, which may be used to create sub-interpreters with their own GILs. (See PEP 684: A Per-Interpreter GIL for more info.) (Contributed by Eric Snow in gh-104110.)

• In the limited C API version 3.12, Py_INCREF() and Py_DECREF() functions are now implemented as opaque function calls to hide implementation details. (Contributed by Victor Stinner in gh-105387.)