Espressif: Update to ESP-IDF v5.4.1

[dhalbert]

Espressif: Update to ESP-IDF v5.4

[AbortRetryFail]

I encountered build problems with the espressif port this morning. It looks like ESP-IDF's tools include overcomplicated fragile Python dependency check scripts. Is this related?

$ make -j6 BOARD=lilygo_tdeck
- Verbosity options: any combination of "steps commands rules", as `make V=...` or env var BUILD_VERBOSE
-- Building ESP-IDF components for target esp32s3
-- Checking Python dependencies...
The following Python requirements are not satisfied:
Requirement 'setuptools<71.0.1,>=21' was not met. Installed version: 78.1.0
To install the missing packages, please run "install.sh"
Diagnostic information:
    IDF_PYTHON_ENV_PATH: /home/abortretryfail/.espressif/python_env/idf5.3_py3.13_env
    Python interpreter used: /home/abortretryfail/.espressif/python_env/idf5.3_py3.13_env/bin/python
Constraint file: /home/abortretryfail/.espressif/espidf.constraints.v5.3.txt
Requirement files:
 - /home/abortretryfail/software/circuitpython/ports/espressif/esp-idf/tools/requirements/requirements.core.txt
Python being checked: /home/abortretryfail/.espressif/python_env/idf5.3_py3.13_env/bin/python
CMake Error at esp-idf/tools/cmake/build.cmake:368 (message):
  Failed to run Python dependency check.  Python: python, Error: 255
Call Stack (most recent call first):
  esp-idf/tools/cmake/build.cmake:502 (__build_check_python)
  esp-idf/tools/cmake/project.cmake:710 (idf_build_process)
  CMakeLists.txt:12 (project)


-- Configuring incomplete, errors occurred!
make: *** [Makefile:559: build-lilygo_tdeck/esp-idf/config/sdkconfig.h] Error 1

A temporary fix was pip install 'cryptography<43' 'setuptools<71.0.1

[eightycc]

@AbortRetryFail There are known python package version conflicts between CircuitPython and ESP-IDF. In a nutshell, you'll need to run esp-idf/install.sh a second time after you've pip'ed the CircuitPython dependencies. Here's my recipe for building Espressif: https://gist.github.com/eightycc/197a12ffb747e21aa72bdedb2cade818

[eightycc]

ESP-IDF 5.4.1 is the current release version. If there's no objection, I'll fit our local commits onto upstream tag v5.4.1.

[tannewt]

Yes, please rebase changes on https://github.com/adafruit/esp-idf/tree/circuitpython-v5.3.2 onto a new circuitpython-v5.4.1 branch. I'll invite you as a repo collaborator there.

[eightycc]

ESP-IDF does more strict checking for valid use of CONFIG_ flags. It now checks that all used flags are defined somewhere in the chain of Kconfig files for the target processor. This applies not just to flags that are defined, but also to flags that are commented out. Although CP does not use Kconfig, these checks are still performed. CP config files all pass, except when building for the ESP32H2. CP configures this for a 48 MHz flash spi clock, but according to Kconfig this isn't a valid setting. The valid setting are 16, 32, and 64. Is this a typo in CP or a setting we want?

I've added support to ESP32H2 Kconfig for a 48 MHz clock.

[eightycc]

Symbol ESP_ROM_ELF_DIR is not getting set, resulting in CI failure on all ESP boards. Oddly, this error does not occur building on my local machine. This is a known problem, see espressif/esp-idf#15035.

[eightycc]

The symbol __atomic_test_and_set is missing in ESP-IDF v5.4.1 causing several boards to fail to build. This is a known regression in v5.4: espressif/esp-idf#15167. Will workaround by adding the commit referenced by the issue to branch circuitpython-v5.4.1 of our ESP-IDF fork.

[eightycc]

With a DEBUG=1 build flashed onto a Feather Huzzah ESP32, double faulting early in startup:

configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:6592
load:0x40078000,len:16684
load:0x40080400,len:4
load:0x40080404,len:4268
entry 0x40080658
I (29) boot: ESP-IDF v5.4.1-6-gf50ec8ecdb 2nd stage bootloader
I (29) boot: compile time Apr 21 2025 09:39:10
I (29) boot: Multicore bootloader
I (32) boot: chip revision: v3.0
I (35) qio_mode: Enabling default flash chip QIO
I (39) boot.esp32: SPI Speed      : 80MHz
I (43) boot.esp32: SPI Mode       : QIO
I (46) boot.esp32: SPI Flash Size : 4MB
I (50) boot: Enabling RNG early entropy source...
I (54) boot: Partition Table:
I (57) boot: ## Label            Usage          Type ST Offset   Length
I (63) boot:  0 nvs              WiFi data        01 02 00009000 00005000
I (70) boot:  1 otadata          OTA data         01 00 0000e000 00002000
I (76) boot:  2 ota_0            OTA app          00 10 00010000 00200000
I (83) boot:  3 user_fs          Unknown data     01 81 00210000 001f0000
I (89) boot: End of partition table
I (93) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=4ba18h (309784) map
I (179) esp_image: segment 1: paddr=0005ba40 vaddr=3ff80060 size=0001ch (    28) load
I (180) esp_image: segment 2: paddr=0005ba64 vaddr=3ffb0000 size=045b4h ( 17844) load
I (188) esp_image: segment 3: paddr=00060020 vaddr=400d0020 size=120420h (1180704) map
I (492) esp_image: segment 4: paddr=00180448 vaddr=3ffb45b4 size=00904h (  2308) load
I (493) esp_image: segment 5: paddr=00180d54 vaddr=40080000 size=17d8ch ( 97676) load
I (526) esp_image: segment 6: paddr=00198ae8 vaddr=400c0000 size=00060h (    96) load
I (526) esp_image: segment 7: paddr=00198b50 vaddr=50000ff0 size=00c00h (  3072) load
I (543) boot: Loaded app from partition at offset 0x10000
I (543) boot: Disabling RNG early entropy source...
I (553) cpu_start: Multicore app
I (561) cpu_start: Pro cpu start user code
I (561) cpu_start: cpu freq: 240000000 Hz
I (561) app_init: Application information:
I (561) app_init: Project name:     circuitpython
I (566) app_init: App version:      10.0.0-alpha.2-79-g22519d88b8
I (572) app_init: Compile time:     Apr 21 2025 09:39:03
I (577) app_init: ELF file SHA256:  19ff33c14...
I (581) app_init: ESP-IDF:          v5.4.1-6-gf50ec8ecdb
I (586) efuse_init: Min chip rev:     v0.0
I (590) efuse_init: Max chip rev:     v3.99 
I (594) efuse_init: Chip rev:         v3.0
I (598) heap_init: Initializing. RAM available for dynamic allocation:
I (604) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (609) heap_init: At 3FFBE418 len 00021BE8 (134 KiB): DRAM
I (614) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (620) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (625) heap_init: At 40097D8C len 00008274 (32 KiB): IRAM
Guru Meditation Error: Core  0 panic'ed (Double exception). 

Core  0 register dump:
PC      : 0x40097c3a  PS      : 0x00040536  A0      : 0x80058d65  A1      : 0x3ffe3ad0  
A2      : 0x00000001  A3      : 0x00000003  A4      : 0x00000000  A5      : 0x00000000  
A6      : 0x00000000  A7      : 0x00000000  A8      : 0x00000001  A9      : 0x00000000  
A10     : 0x00000000  A11     : 0x00000003  A12     : 0x3ffe3ad0  A13     : 0x3f41489c  
A14     : 0x3ffb9b64  A15     : 0x3ffae97c  SAR     : 0x0000001f  EXCCAUSE: 0x00000002  
EXCVADDR: 0x3f41488c  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  


Backtrace: 0x40097c37:0x3ffe3ad0 0x40058d62:0x3ffe3b30 0x40058d62:0x00000000 |<-CORRUPTED




ELF file SHA256: 19ff33c14

CPU halted.

[dhalbert]

With a DEBUG=1 build flashed onto a Feather Huzzah ESP32, double faulting early in startup:

I am getting the same problem on the tip of main for Feather Huzzah32, so this appears not to be a v5.4.1 problem. I'll do a bisect between there and alpha.2, which works OK.

[dhalbert]

With a DEBUG=1 build flashed onto a Feather Huzzah ESP32, double faulting early in startup:

I am getting the same problem on the tip of main for Feather Huzzah32, so this appears not to be a v5.4.1 problem. I'll do a bisect between there and alpha.2, which works OK.

It turned out this double faulting crash only occurs with DEBUG=1 on main and 10.0.0-alpha.2 for Feather Huzzah ESP32. It also occurs with a DEBUG=1 build on main and 10.0.0-alpha.2 are fine without DEBUG=1.