gh-119842: Honor PyOS_InputHook in the new REPL

[pablogsal]

• Issue: Honor PyOS_InputHook in the new REPL #119842

[mdboom]

Unfortunately, this doesn't seem to address the bug and restore old behavior, see #119842 (comment)

[mdboom]

Since Tkinter's PyOS_InputHook implementation sets the thread state, we need to ALLOW_THREADS around it. We could either do that in Tkinter (but that would be a new requirement on all PyOS_InputHook implementations in the wild), or instead call PyOS_InputHook through a wrapper that allows threads. I have a proof of concept here to show what I mean. This seems to make matplotlib and the simple Tkinter reproducer work as expected (events are handled, the window repaints and resizes etc.)

I share @ambv's concern that this isn't called periodically, but at least for Tkinter it doesn't matter. Maybe it does for other GUI toolkits, though -- it looks like PyQt5 does, for example.

[pablogsal]

or instead call PyOS_InputHook through a wrapper that allows threads.

I think that's our best approach here because it was previously readline dropping the GIL around it so now is us that need to do it and we must to that from C.

[pablogsal]

Ok, I think I have a version that should be backwards compatible with the periodic calls that readline.c does. It may be still have some rough edges but I want to discuss before doing more with this. @ambv @mdboom

[tacaswell]

Things that may be useful references:

• IPython's inputhooks (for prompt toolkit) https://github.com/ipython/ipython/tree/main/IPython/terminal/pt_inputhooks
• ipykernel event loop hooks (for tornado, it is a web server, but same theory) https://github.com/ipython/ipykernel/blob/main/ipykernel/eventloops.py
• https://github.com/matplotlib/matplotlib/blob/a833d99e46f5771aee7c25066276a70e6e7b9ef3/src/_macosx.m#L54-L96 / https://github.com/matplotlib/matplotlib/blob/a833d99e46f5771aee7c25066276a70e6e7b9ef3/src/_macosx.m#L191-L202 mpl's PyOS_InptuHook for the macOS UI
• pyqt5 and pyqt6 have implementations, but not sure if you want to look at them due to GPL issues
• pyside's proposed patch (which I am not 100% sure is correct) https://codereview.qt-project.org/c/pyside/pyside-setup/+/454254/2/sources/pyside6/libpyside/pyside.cpp
• Matplotlib's documentation on input hooks https://matplotlib.org/stable/users/explain/figure/interactive_guide.html#input-hook-integration (if these are wrong I'll fix them)
• discussion of wxPython https://discuss.wxpython.org/t/interactive-usage-of-wxpython/32128/22 but this seems to be lost in the current version (I can not find PyOS_InputHook in the current source and mpl does not work with py312 @DietmarSchwertberger (who has worked on both mpl and wxpython) or @RobinD42 may be able to comment on that?
• old bug reports about gtk using PyOS_InputHook (https://stackoverflow.com/questions/32754630/runtimewarning-pyos-inputhook-is-not-available-for-interactive-use-of-pygtk ) but I can not find it in their current source (but also not sure I am even looking at the right place...I do not undrestand how pygobject works) and mpl does not work with py312. I did find https://github.com/nzjrs/pygtk/blob/5bee5a7850bbe8ce0581afe8b9a71eaa52d7206b/gtk/gtk.override#L457-L480 but that repo was last touched 13 years ago and looks like a soft-fork of something upstream
• https://github.com/search?q=PyOS_InputHook&type=code&p=1 has a lot of hits

The best-case behavior is when the input hook can watch stdin and exit the event loop when there is input to read, second best case is the hook run the loop for some short period and checks if there is input in a hotloop.

Mike is right, the expect expectation is that the window is reasonably responsive (<100ms) and you can type at the prompt (and it is also responsive with tab-complete etc). In general the time scale we need to switch between the two loops is human interaction time so it is OK if the prompt does not work while the user is interacting with the GUI so long as the prompt gets control again by the time the user can change their "focus" there (e.g. move mouse and click, then type).

I suspect that I'm going to be making some upstream bug reports to gtk and wx after this...

Taking a very selfish (from Matplotlib's point of view) position, if you want to re-design the input hook so long as tkinter (which I assume you will do ;) ), riverside (for PyQt) (worst case I can try to put in a patch), and Matplotlib (for macos which I can make sure gets done) provide the new hook we will get back to status quo.

[mdboom]

This looks much closer to the original semantics, so I'm a lot more confident this behavior matches the legacy REPL.

I volunteered to do some testing, but I'm starting to think it isn't worth the effort to be extremely comprehensive (unless others strongly disagree).

• I did test Tkinter on Windows and Linux, and it's working there -- and it's easy to test by creating a window and confirming it's resizable and responsive.
• matplotlib has a PyOS_InputHook for the macOS Cocoa windowing toolkit -- that should be easy enough to test, but I don't have a Mac.
• Other GUI frameworks (the important ones are probably PySide6, pygobject-gtk and wxPython) don't use PyOS_InputHook directly, however in the common case, IPython installs input hooks for them. But with IPython, I think by definition aren't using the new CPython REPL anyway so that's a non-issue. (i.e. the input hooks don't usually exist for the default legacy CPython REPL, so if they are broken then, they are still broken in the same way with the new CPython REPL).

[tacaswell]

It is probably worth testing PyQt5 and PyQt6 (I do not have time to do this today).

[ambv]

I confirm this PR also fixes the matplotlib example given by Michael in the issue on macOS Sonoma.

[ambv]

PyQt is important to us. We'll test it and adjust the behavior if needed. Now I need to land this to make it into 3.13 beta 2.

[miss-islington-app]

Thanks @pablogsal for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @pablogsal and @ambv, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker d9095194dde27eaabfc0b86a11989cdb9a2acfe1 3.13

[ambv]

I'll backport this myself once gh-120062 lands.

[bedevere-app]

GH-120066 is a backport of this pull request to the 3.13 branch.

[tacaswell]

Thank you @pablogsal @ambv and @mdboom !