[BUG] source scripts/bootstrap.sh fails when using Python 3.11 (Debian Bookworm)

[s-marios]

Reproduction steps

On Debian Bookworm, python ver. 3.11.2, the bootstrap script fails.

To reproduce, follow the "building matter" document,

0. install prerequisites
1. git clone --recurse-submodules
2. source scripts/bootstrapt.sh

This resulted in the log file attached. console_output.txt

In sort, it fails to build (or actually install?) the wheels for gevent and greenlet.

I have no idea why this happens or how I could proceed from there.

Bug prevalence

Whenever I do this

GitHub hash of the SDK that was being used

e75d35a

Platform

core

Platform Version(s)

No response

Anything else?

This change in Debian about having python interpreters marked as externally managed may be of relevance.

Also, sometime in December I had no problem with the bootstrap script and was in fact able to build matter successfully. However, it seems that further updates in Debian Testing broke the bootstrap script.

Using a venv (pyton3.11 -m venv chip-venv) led to the same result.

Can you please look into this? Any insights are appreciated.

[s-marios]

Small update, trying the procedure in a Debian Stable lxc container works as expected, so Debian Testing is definitely the suspect here.

[bzbarsky-apple]

@andy31415 @mspang

[andy31415]

× Building wheel for gevent (pyproject.toml) did not run successfully.

We seem to have had this problem occasionally due to esp32 requirement (gdbui -> gevent). @dhrishi ... any chance we can avoid esp32 requiring gdbgui?

@s-marios could you test out what happens if you edit scripts/setup/requirements.esp32.txt and comment out (place a # in from of) the gdbgui line?

[s-marios]

@s-marios could you test out what happens if you edit scripts/setup/requirements.esp32.txt and comment out (place a # in from of) the gdbui line?

This fixes things and activate.sh/bootstrap.sh seem to work.

I guess this does not affect normal development and the only thing that we miss out on is being able to use gdbui, am I right?

[mspang]

Do we perhaps just need to update the constraints.txt ?

[mspang]

Doesn't seem like there is a newer version of gevent, but I posted a PR to update anyway since it hasn't been done in a while:

#25441

[mspang]

greenlet was updated. It's still possible this will resolve your issue. Mind trying it @s-marios ?

[andy31415]

if we don't install gdbui I believe esp32 builds may fail because it is mandatory there (builds switch to chip venv and then builds may completely fail, even if idl exports.sh would install it).

If we make changes like commenting out gdbui, we have to ensure esp32 builds work (generally CI would check that, so that may be sufficient)

[andy31415]

Tried the fix in 25441 on my arch machine (which has the gdbui issue), no help unfortunately:

...
      src/gevent/greenlet.c: In function ‘__pyx_f_6gevent_9_greenlet_get_generic_parent’:
      src/gevent/greenlet.c:16215:26: error: ‘PyGreenlet’ {aka ‘struct _greenlet’} has no member named ‘parent’
      16215 |   __pyx_t_1 = ((__pyx_v_s->parent != NULL) != 0);
...

[andy31415]

Just checked in our latest image (with latest esp32) and gdbui is NOT in /opt/espressif/esp-idf/requirements.txt
maybe just removing is acceptable after all.

[andy31415]

I was missing a g in my grep. It is there after all as gdbgui==0.13.2.0 :(

We could patch it in as part of our build image building, however that is probably not maintainable. I would much rather have esp-idf updated. @dhrishi

[mspang]

Ok, well, it sounds like this may be blocked on resolving some upstream dependency issues.

In order to fix compatibility issues, the dependencies must make a new release. So they must be updated. But we are blocking a handful of updates (presumably due to some breakage)

grep '^[^ ]*\(==\|<\)' requirements.*
requirements.esp32.txt:pyparsing>=2.0.3,<2.4.0
requirements.esp32.txt:pygdbmi<=0.9.0.2
requirements.esp32.txt:reedsolo>=1.5.3,<=1.5.4
requirements.esp32.txt:kconfiglib==13.7.1
requirements.esp32.txt:construct==2.10.54
requirements.esp32.txt:python-socketio<5
requirements.esp32.txt:gdbgui==0.13.2.0 ; platform_machine != 'aarch64' and sys_platform == 'linux'
requirements.esp32.txt:jinja2<3.1
requirements.esp32.txt:itsdangerous<2.1
requirements.mbed.txt:pytest==6.2.5 ; platform_machine != 'aarch64' and sys_platform == 'linux'
requirements.mbed.txt:mbed-ls==1.8.11 ; platform_machine != 'aarch64' and sys_platform == 'linux'
requirements.telink.txt:future==0.18.3
requirements.telink.txt:pypng==0.0.21
requirements.telink.txt:PyQRCode==1.2.1
requirements.txt:build==0.8.0
requirements.txt:mypy==0.971
requirements.txt:mypy-protobuf==3.2.0
requirements.txt:protobuf==3.20.1
requirements.txt:types-protobuf==3.19.22

Trying to hold these back forever won't work, as the configuration eventually becomes unsatisfiable.

[andy31415]

Just removing gdbgui will not work:

# ./scripts/build/build_examples.py --target esp32-m5stack-all-clusters build
...
2023-03-02 21:30:35 INFO    Environment looks good, you are ready to go!
2023-03-02 21:30:35 INFO
2023-03-02 21:30:35 WARNING The following Python requirements are not satisfied:
2023-03-02 21:30:35 WARNING gdbgui==0.13.2.0
2023-03-02 21:30:35 WARNING To install the missing packages, please run "/opt/espressif/esp-idf/install.sh"
2023-03-02 21:30:35 WARNING Diagnostic information:
2023-03-02 21:30:35 WARNING     IDF_PYTHON_ENV_PATH: /opt/espressif/tools/python_env/idf4.4_py3.9_env
2023-03-02 21:30:35 WARNING     Python interpreter used: /home/vscode/pigweed/env/pigweed-venv/bin/python
....

[s-marios]

if we don't install gdbui I believe esp32 builds may fail because it is mandatory there (builds switch to chip venv and then builds may completely fail, even if idl exports.sh would install it).

That's unfortunate, as it also happens that I'm targeting esp32.

Tried the fix in 25441 on my arch machine (which has the gdbui issue)

So, this is not a Debian problem, it seems to be a python 3.11 problem I suppose? If that's the case, let's change the title of this issue to something more appropriate.