(dev): 3.11 testing

[Skylion007]

Description

• Testing 3.11

Suggested changelog entry:

* Fix issues and test against CPython 3.11 betas.

[Skylion007]

@rwgk Any ideas how to address this failure? I would like to get PyBind11 fully supported before 3.11 leaves alpha.

[rwgk]

@rwgk Any ideas how to address this failure? I would like to get PyBind11 fully supported before 3.11 leaves alpha.

No, unfortunately not. First thought: this could be really tricky to debug and fix. MRO is complex, especially between native Python & pybind11:

https://pybind11.readthedocs.io/en/stable/advanced/classes.html

... but once you begin mixing Python and C++ multiple inheritance, things will fall apart due to differences between Python’s MRO and C++’s mechanisms.

That's all I know though, I don't have first-hand experience working on the MI code.

What I'd try first: comment out tests until everything that's left passes. Then look at the overall picture to decide what to do next.

[Skylion007]

@rwgk I know what causes the issue. It's a corner case caused when a PyBind11 Class has multiple inheritance and the first base does not have dynamic-attrs, but another base class does not.

[rwgk]

I only remembered that very vaguely.

Variation of my "comment out" strategy: manually add py::dynamic_attr to see if that makes everything pass. If that's do-able: could that even be the final solution, maybe with a better error message?

[Skylion007]

@rwgk Adding it to the children classes did not help. Adding it to the base classes does work, but then that makes this unit test stops doing anything meaningful and the base class and any of its parents inherit the performance penalty of the dynamic-attrs.

[rwgk]

@rwgk Adding it to the children classes did not help. Adding it to the base classes does work, but then that makes this unit test stops doing anything meaningful and the base class and any of its parents inherit the performance penalty of the dynamic-attrs.

Sound fine to me:

• Most importantly, everything works. (I guess)
• In all likelihood, the performance penalty is inconsequential. (If not, they have a bigger problem IMO.)
• If someone disagrees regarding the performance penalty, invite them to work out a fix.

My net conclusion: Add py::dynamic_attrs to the base class with a terse comment and link pointing here.

[henryiii]

FYI, beta1 is out, no more non-bug fix changes.

[vstinner]

In the Python 3.10 code path, PySys_SetArgvEx() is called after Python initialization, when sys.path was already computed. In this case, the function inserts a new path at sys.path[0]:

If argv[0] is not '-c' nor '-m', prepend argv[0] to sys.path.

If you don't call PySys_SetArgvEx(), Py_RunMain() inserts a new path at sys.path[0] if safe_path=0: the path depends on PyConfig.argv[0].

I tried to document everything about the Python Path Configuration at: https://docs.python.org/dev/c-api/init_config.html#python-path-configuration See the "Py_RunMain() and Py_Main() modify sys.path: (...)" section.

If you control argv and you know what should be in sys.argv[0], you can just modify sys.path manually. Dummy example without error handling:

PyRun_SimpleString("import sys; sys.path.insert(0, '/my/path')");

Note: the code to compute sys.path has been rewritten in pure Python in Python 3.11: https://github.com/python/cpython/blob/main/Modules/getpath.py But this code doesn't include changes done afterwards by Py_RunMain(), as I wrote.

[rwgk]

Looks like Henry was faster by a few minutes.

[rwgk]

I'm really concerned about inlining the two complex functions here, especially because each version introduces multiple variables which could clash in non-obvious way for one Python version or another. We don't know how this code will need to be evolved in the future, accidents seem likely. I think it would be much better to keep the functions separate, even if we need #ifdef in both places, above for the function definitions, and here (I didn't look enough to know).

[rwgk]

Consistency would be nice, e.g.

3.11-latest above (line 33) and here.

I often look at CI logs in bulk, having to manually map latest internals here to -dev elsewhere is confusing/distracting.

[rwgk]

A comment here would be useful (similar to Henry's comment from a few days ago):

• Link to the corresponding Python sources.
• Something like "unfortunately the Python code is private, therefore we are forced to duplicate here".

[vstinner]

What has to be duplicated?

[rwgk]

I had this comment in mind:

#3923 (comment)

@henryiii wrote 14 days ago:

_PyPathConfig_ComputeSysPath0 is private. :( It seems looking at PEP 587 that all the sys.path handling is now tied to the high level API (Py_Main, Py_BytesMain, and Py_RunMain), and line-by-line runners like us are stuck reimplementing the logic ourselves now.

When I saw the email that Henry had already merged this PR I stopped looking further. Doing that now:

This seems to be what @henryiii was referring to (and the "Link" I had in mind):

• https://github.com/python/cpython/blob/29f86d6c28fa9ed39e42678776d6005eaf5090c8/Python/pathconfig.c#L388

Comparing that with the new #else (Python 3.11) block here in embed.h, it looks like something completely different. Apparently I misinterpreted Henry's comment. Sorry.

Looking more, it seems this is @vstinner's response to the @henryiii's comment above:

• (dev): 3.11 testing #3923 (comment)

Which makes me think adding this link would be useful:

    if (add_program_dir_to_path) {
            // https://docs.python.org/dev/c-api/init_config.html#python-path-configuration
            PyRun_SimpleString("import sys, os.path; "
                           "sys.path.insert(0, "
                           "os.path.abspath(os.path.dirname(sys.argv[0])) "
                           "if sys.argv and os.path.exists(sys.argv[0]) else '')");
    }

Or maybe alternatively the link to Victor's comment.

[vstinner]

PyConfig API initializes the default value of sys.path. But Py_RunMain() then inserts a path to sys.path. Python 3.11 safe_path=1 disables this behavior:

• https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.safe_path
• https://docs.python.org/dev/c-api/init_config.html#python-path-configuration

It might be interesting to move more code changing sys.path into the Python initialization (PyConfig), but it's complicated.