Compatibility with Python3.11 (again)

[tacaswell]

python/cpython#30590 made changes to the way exceptions are stored on the thread stucts and greenlet does not compile

$ gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleedin
g/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::ExceptionState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:640:31: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  640 |     this->exc_state = tstate->exc_state;
      |                               ^~~~~~~~~
      |                               _exc_state
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::ExceptionState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:645:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  645 |     tstate->exc_state = this->exc_state;
      |             ^~~~~~~~~
      |             _exc_state
src/greenlet/greenlet_greenlet.hpp:647:52: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_state’; did you mean ‘_exc_state’?
  647 |         this->exc_info ? this->exc_info : &tstate->exc_state;
      |                                                    ^~~~~~~~~
      |                                                    _exc_state
src/greenlet/greenlet_greenlet.hpp: In constructor ‘greenlet::PythonState::PythonState()’:
src/greenlet/greenlet_greenlet.hpp:797:42: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘root_cframe’; did you mean ‘_root_cframe’?
  797 |     this->cframe = &PyThreadState_GET()->root_cframe;
      |                                          ^~~~~~~~~~~
      |                                          _root_cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_new_cframe(CFrame&)’:
src/greenlet/greenlet_greenlet.hpp:919:52: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘root_cframe’; did you mean ‘_root_cframe’?
  919 |     this->cframe->previous = &PyThreadState_GET()->root_cframe;
      |                                                    ^~~~~~~~~~~
      |                                                    _root_cframe

[blackPantherOS]

Python-3.11
Gcc-11.2.1
Cython: 0.29.26

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:830:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  830 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
src/greenlet/greenlet_greenlet.hpp:831:36: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  831 |     this->_top_frame.steal(tstate->frame);
      |                                    ^~~~~
      |                                    cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:861:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  861 |     tstate->frame = this->_top_frame.relinquish_ownership();
      |             ^~~~~
      |             cframe
src/greenlet/greenlet_greenlet.hpp:862:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  862 |     tstate->recursion_depth = this->recursion_depth;
      |             ^~~~~~~~~~~~~~~
      |             recursion_limit
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_initial_state(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:883:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  883 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
error: command '/usr/bin/gcc' failed with exit code 1

[brandtbucher]

We went ahead and reverted that change:

python/cpython#31038

[ericsnowcurrently]

We went ahead and reverted that change:

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

[brandtbucher]

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

That was part of my original fix in #280. I don't believe @blackPantherOS's failure is happening on Greenlet's master branch anymore.

[tacaswell]

At least locally for me it compiles and python -m unittest discover -v greenlet.tests reports success.

[blackPantherOS]

I only fixed root_cframe and exc_state in python/cpython#31038. recursion_depth was changed in GH-29524 and hasn't been sorted out, so it is probably still breaking greenlet.

That was part of my original fix in #280. I don't believe @blackPantherOS's failure is happening on Greenlet's master branch anymore.

Thanks, I tried latest git (2.0.0a1) as well

it clone https://github.com/python-greenlet/greenlet.git
Cloning into 'greenlet'...
remote: Enumerating objects: 3914, done.
remote: Counting objects: 100% (1804/1804), done.
remote: Compressing objects: 100% (446/446), done.
remote: Total 3914 (delta 1181), reused 1671 (delta 1085), pack-reused 2110
Receiving objects: 100% (3914/3914), 1.35 MiB | 4.15 MiB/s, done.
Resolving deltas: 100% (2530/2530), done.
└──╼ # cd greenlet/
└──╼ # python3 set
setup.cfg  setup.py   
└──╼ # python3 setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.11
creating build/lib.linux-x86_64-3.11/greenlet
[...]

gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -O2 -g -pipe -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -fPIC -I/usr/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:830:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  830 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
src/greenlet/greenlet_greenlet.hpp:831:36: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  831 |     this->_top_frame.steal(tstate->frame);
      |                                    ^~~~~
      |                                    cframe
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:861:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
  861 |     tstate->frame = this->_top_frame.relinquish_ownership();
      |             ^~~~~
      |             cframe
src/greenlet/greenlet_greenlet.hpp:862:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  862 |     tstate->recursion_depth = this->recursion_depth;
      |             ^~~~~~~~~~~~~~~
      |             recursion_limit
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::set_initial_state(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:883:37: error: ‘const PyThreadState’ {aka ‘const struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
  883 |     this->recursion_depth = tstate->recursion_depth;
      |                                     ^~~~~~~~~~~~~~~
      |                                     recursion_limit
error: command '/usr/bin/gcc' failed with exit code 1

[tacaswell]

I am now seeing a different failure

$ gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.11 -c src/greenlet/greenlet.cpp -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
In file included from src/greenlet/greenlet_internal.hpp:19,
                 from src/greenlet/greenlet.cpp:17:
src/greenlet/greenlet_greenlet.hpp:137:9: error: ‘CFrame’ does not name a type; did you mean ‘_frame’?
  137 |         CFrame* cframe;
      |         ^~~~~~
      |         _frame
src/greenlet/greenlet_greenlet.hpp:142:9: error: ‘_interpreter_frame’ does not name a type; did you mean ‘_PyInterpreterFrame’?
  142 |         _interpreter_frame *current_frame;
      |         ^~~~~~~~~~~~~~~~~~
      |         _PyInterpreterFrame
src/greenlet/greenlet_greenlet.hpp:163:29: error: ‘CFrame’ has not been declared
  163 |         void set_new_cframe(CFrame& frame) G_NOEXCEPT;
      |                             ^~~~~~
src/greenlet/greenlet_greenlet.hpp: In constructor ‘greenlet::PythonState::PythonState()’:
src/greenlet/greenlet_greenlet.hpp:736:6: error: class ‘greenlet::PythonState’ does not have any field named ‘cframe’
  736 |     ,cframe(nullptr)
      |      ^~~~~~
src/greenlet/greenlet_greenlet.hpp:741:6: error: class ‘greenlet::PythonState’ does not have any field named ‘current_frame’
  741 |     ,current_frame(nullptr)
      |      ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp:797:11: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  797 |     this->cframe = &PyThreadState_GET()->root_cframe;
      |           ^~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator<<(const PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:817:11: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  817 |     this->cframe = tstate->cframe;
      |           ^~~~~~
src/greenlet/greenlet_greenlet.hpp:822:11: error: ‘class greenlet::PythonState’ has no member named ‘current_frame’
  822 |     this->current_frame = tstate->cframe->current_frame;
      |           ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: In member function ‘void greenlet::PythonState::operator>>(PyThreadState*)’:
src/greenlet/greenlet_greenlet.hpp:844:28: error: ‘class greenlet::PythonState’ has no member named ‘cframe’
  844 |     tstate->cframe = this->cframe;
      |                            ^~~~~~
src/greenlet/greenlet_greenlet.hpp:855:43: error: ‘class greenlet::PythonState’ has no member named ‘current_frame’
  855 |     tstate->cframe->current_frame = this->current_frame;
      |                                           ^~~~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp: At global scope:
src/greenlet/greenlet_greenlet.hpp:910:6: error: variable or field ‘set_new_cframe’ declared void
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |      ^~~~~~~~~~~
src/greenlet/greenlet_greenlet.hpp:910:34: error: ‘CFrame’ was not declared in this scope; did you mean ‘_frame’?
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |                                  ^~~~~~
      |                                  _frame
src/greenlet/greenlet_greenlet.hpp:910:42: error: ‘frame’ was not declared in this scope; did you mean ‘_frame’?
  910 | void PythonState::set_new_cframe(CFrame& frame) G_NOEXCEPT
      |                                          ^~~~~
      |                                          _frame
src/greenlet/greenlet.cpp: In member function ‘virtual greenlet::Greenlet::switchstack_result_t greenlet::UserGreenlet::g_initialstub(void*)’:
src/greenlet/greenlet.cpp:1037:5: error: ‘CFrame’ was not declared in this scope; did you mean ‘_frame’?
 1037 |     CFrame trace_info;
      |     ^~~~~~
      |     _frame
src/greenlet/greenlet.cpp:1039:39: error: ‘trace_info’ was not declared in this scope
 1039 |     this->python_state.set_new_cframe(trace_info);
      |                                       ^~~~~~~~~~

I believe (but have not debugged/proven) that this is related to 18b5dd68c6b616257ae243c0b6bb965ffc885a23 / python/cpython#31530 / https://bugs.python.org/issue46836 which moved around some of the (internal) structs.

[brandtbucher]

Thanks for the update.

I'm working on seeing if we can revert those changes in CPython, rather than requiring another layer of compatibility macros here.

[tacaswell]

When these thing happen I should start with greenlet as other projects seem happier to adapt!

This is a very late dependency in my personal stack so I tend to get to it after dealing with cython/numpy/scipy/pybind11 etc.

[brandtbucher]

When these thing happen I should start with greenlet as other projects seem happier to adapt!

I'm not affiliated with Greenlet, but my team is interested in keeping a few projects like Greenlet working for dev versions of CPython (since we need them to benchmark our performance work).

[zzzeek]

Hi Greenlet -

This is really not my usual style as I am very familiar with people poking me about issues that will be fixed, "when they're fixed", but Python 3.11 is at 3.11b1 now, downstream distros are packaging things and being able to build greenlet on py311 is starting to really hold things up. Is there any workaround / patch / specific tag available so that downstream dependents like SQLAlchemy can move forward with Py 3.11? thanks for listening.

• mike

[zzzeek]

looks like #302 does it, thanks @tacaswell