Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1

[brentru]

@maxgerhardt @ivankravets Hi, I was wondering if there are any plans to synchronize the development branch of this repository with the IDF v5.1 alpha releases? I understand that these releases are in the alpha stage, but I believe that updating the PIO package to be compatible with IDF v5+ could greatly benefit the community.

If this isn't in the roadmap, could you please suggest any potential workarounds?

Arduino Alpha 2 Release v3.0.0 based on ESP-IDF v5.1

[maxgerhardt]

Can't speak for PIO Labs as a company since I'm not even an employee, but I've read multiple times that PIO's policy is to support stable versions, not alpha / beta versions. So if it's released as stable, support will probably follow.

Nevertheless, @Jason2866 has a Arduino/IDF5 branch in his https://github.com/Jason2866/platform-espressif32. So your first try should be to just use

platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5

in your platformio.ini and see where that takes you.

[mhaberler]

@maxgerhardt thanks!

I tried your suggestion with a small esp32s3 project with Arduino as an esp-idf component using PlatformIO 6.1.11

platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5
framework = arduino, espidf

build and running works fine

what does not work any more is JTAG debugging as gdb is not found

FileNotFoundError: [Errno 2] No such file or directory: 'xtensa-esp32s3-elf-gdb'

I guess this has to do with the factoring out of esp32 gdb into its own repo, it is not present in .platformio/packages/toolchain-xtensa-esp32s3/bin/

I assume this would be available from https://docs.espressif.com/projects/esp-idf/en/v5.1.1/esp32s3/api-guides/tools/idf-tools.html?highlight=xtensa%20esp32s3%20elf%20gdb#xtensa-esp-elf-gdb

but I'm at loss on how to add this to my installation/project?

edit: found #1126 - will try if this works now

edit2: well the replace_gdb.py trick worked so far: #1126 (comment)

[maxgerhardt]

FileNotFoundError: [Errno 2] No such file or directory: 'xtensa-esp32s3-elf-gdb'

You might be able to counteract this by explicitly enabling the package like this does by in the platformio.ini with

platform_packages = tool-xtensa-esp-elf-gdb

If that does not work you might need to patch the ~/.platformio/platforms/espressif32/platform.json to include the declaration of these packages like mainline.

[Jason2866]

The gdb are listed in the platform.json. I will have a look why the are not installed.
My Python Build Scripts differ from the official ones. Probably overlooked to install them...

[Jason2866]

@mhaberler @maxgerhardt Fixed. Thx for the issue info. The gdb packages are now installed.

[mhaberler]

thanks, yes this did have the same effect and got me the proper gdb version:

Processing N8R8 (board: waveshare-ESP32-S3-DEV-KIT-N8R8; platform: https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5; framework: arduino, espidf)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/waveshare-ESP32-S3-DEV-KIT-N8R8.html
PLATFORM: Espressif 32 (2023.8.1+sha.7c70aa1) > Espressif ESP32-S3-WROOM-1U-N8R8
HARDWARE: ESP32S3 240MHz, 512KB RAM, 8MB Flash
DEBUG: Current (esp-builtin) On-board (esp-builtin) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 @ 3.0.0 
 - framework-espidf @ 3.50101.0 (5.1.1) 
 - tool-cmake @ 3.21.3 
 - tool-esptoolpy @ 1.40700.0+dev (4.7.0) 
 - tool-mklittlefs @ 3.2.0 
 - tool-mkspiffs @ 2.230.0 (2.30) 
 - tool-ninja @ 1.9.0 
 - tool-openocd-esp32 @ 2.1100.20220706 (11.0) 
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002 
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0) 
 - toolchain-riscv32-esp @ 12.2.0+20230208 
 - toolchain-xtensa-esp32s3 @ 12.2.0+20230208

overall this looks pretty good so far!

remaining nit: if I enable PSRAM:

CONFIG_ESPTOOLPY_HEADER_FLASHSIZE_UPDATE=y
CONFIG_AUTOSTART_ARDUINO=y
CONFIG_ARDUHAL_LOG_DEFAULT_LEVEL_VERBOSE=y
CONFIG_ARDUHAL_LOG_COLORS=y
CONFIG_ARDUHAL_ESP_LOG=y
CONFIG_COMPILER_STACK_CHECK_MODE_NORM=y
CONFIG_COMPILER_WARN_WRITE_STRINGS=y
CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_SPIRAM_IGNORE_NOTFOUND=y
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
# CONFIG_ESP_INT_WDT is not set
# CONFIG_ESP_TASK_WDT_INIT is not set
CONFIG_FREERTOS_HZ=1000
CONFIG_LOG_DEFAULT_LEVEL_DEBUG=y

the build fails with

Building in debug mode
Compiling .pio/build/N8R8/esp32-hal-psram.o
/Users/mah/.platformio/packages/framework-arduinoespressif32-src-3175c3f96db9cfefc7591c2cd26d1e5d/cores/esp32/esp32-hal-psram.c:22:10: fatal error: esp_psram.h: No such file or directory

*******************************************************************
* Looking for esp_psram.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:esp_psram.h"
* Web  > https://registry.platformio.org/search?q=header:esp_psram.h
*
*******************************************************************

   22 | #include "esp_psram.h"
      |          ^~~~~~~~~~~~~
compilation terminated.

could be a variant issue

I could work around this by -I/Users/mah/.platformio/packages/framework-arduinoespressif32-src-3175c3f96db9cfefc7591c2cd26d1e5d/tools/esp32-arduino-libs/esp32s3/include/esp_psram/include and PSRAM seems to come up:

=========== Before Setup Start ===========
Chip Info:
------------------------------------------
  Model             : ESP32-S3
  Package           : 0
  Revision          : 1
  Cores             : 2
  Frequency         : 240 MHz
  Embedded Flash    : No
  Embedded PSRAM    : No
  2.4GHz WiFi       : Yes
  Classic BT        : No
  BT Low Energy     : Yes
  IEEE 802.15.4     : No
------------------------------------------
INTERNAL Memory Info:
------------------------------------------
  Total Size        :   419843 B ( 410.0 KB)
  Free Bytes        :   355219 B ( 346.9 KB)
  Allocated Bytes   :    59972 B (  58.6 KB)
  Minimum Free Bytes:   350743 B ( 342.5 KB)
  Largest Free Block:   253952 B ( 248.0 KB)
------------------------------------------
SPIRAM Memory Info:
------------------------------------------
  Total Size        :  8388608 B (8192.0 KB)
  Free Bytes        :  8386308 B (8189.8 KB)
  Allocated Bytes   :        0 B (   0.0 KB)
  Minimum Free Bytes:  8386308 B (8189.8 KB)
  Largest Free Block:  8257536 B (8064.0 KB)
  Bus Mode          : OPI
------------------------------------------
Flash Info:
------------------------------------------
  Chip Size         :  8388608 B (8 MB)
  Block Size        :    65536 B (  64.0 KB)
  Sector Size       :     4096 B (   4.0 KB)
  Page Size         :      256 B (   0.2 KB)
  Bus Speed         : 80 MHz
  Bus Mode          : DIO
------------------------------------------
Partitions Info:
------------------------------------------
                nvs : addr: 0x00009000, size:    24.0 KB, type: DATA, subtype: NVS
           phy_init : addr: 0x0000F000, size:     4.0 KB, type: DATA, subtype: PHY
            factory : addr: 0x00010000, size:  1024.0 KB, type:  APP, subtype: FACTORY
------------------------------------------
Software Info:
------------------------------------------
  Compile Date/Time : Oct 19 2023 15:58:40
  ESP-IDF Version   : 5.1.1.231006
  Arduino Version   : 3.0.0
------------------------------------------
Board Info:
------------------------------------------
  Arduino Board     : ESP32S3_DEV
  Arduino Variant   : esp32s3
  Core Debug Level  : 5
  Arduino Runs Core : 1
  Arduino Events on : 1
  Arduino USB Mode  : 1
  CDC On Boot       : 1
============ Before Setup End ============

JTAG debugging (without uploading) works as well

I am impressed @Jason2866 !

thanks @maxgerhardt !

[mhaberler]

got curious what would happen if I slide this underneath m5stack/M5Unified@^0.1.10 and m5stack/M5GFX@^0.1.10 for a m5stack-coreS3 target @lovyan03

there is fallout, but pretty minimal - just two hiccups on 3.0 API changes:

.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp: In member function 'virtual bool lgfx::v1::Light_PWM::init(uint8_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp:43:5: error: 'ledcSetup' was not declared in this scope
   43 |     ledcSetup(_cfg.pwm_channel, _cfg.freq, PWM_BITS);
      |     ^~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp:44:5: error: 'ledcAttachPin' was not declared in this scope; did you mean 'ledcAttach'?
   44 |     ledcAttachPin(_cfg.pin_bl, _cfg.pwm_channel);
      |     ^~~~~~~~~~~~~
      |     ledcAttach
... and ...
*** [.pio/build/coreS3/lib899/M5GFX/lgfx/v1/platforms/esp32/Light_PWM.cpp.o] Error 1
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp: In function 'void lgfx::v1::calcClockDiv(uint32_t*, uint32_t*, uint32_t*, uint32_t*, uint32_t, uint32_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp:167:28: warning: conversion from 'float' to 'uint32_t' {aka 'long unsigned int'} may change value [-Wfloat-conversion]
  167 |         uint32_t b = roundf(fdiv * a);
      |                      ~~~~~~^~~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp:171:35: warning: conversion from 'float' to 'uint32_t' {aka 'long unsigned int'} may change value [-Wfloat-conversion]
  171 |         uint32_t freq = baseClock / ((n * cnt) + (float)(b * cnt) / (float)a);
      |                         ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Bus_SPI.cpp: In function 'void lgfx::v1::gpio_reset(size_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Bus_SPI.cpp:182:5: error: 'gpio_matrix_out' was not declared in this scope; did you mean 'gpio_iomux_out'?
  182 |     gpio_matrix_out((gpio_num_t)pin, SIG_GPIO_OUT_IDX, 0, 0);
      |     ^~~~~~~~~~~~~~~
      |     gpio_iomux_out
*** [.pio/build/coreS3/lib899/M5GFX/lgfx/v1/platforms/esp32/Bus_SPI.cpp.o] Error 1

[maxgerhardt]

I think the LEDC API have changed per this so you might be able to fork the library and correct their source code, with some

#if ESP_ARDUINO_VERSION_MAJOR >= 3
/* code */ 
#else 
/* code */
#endif

check around that code to still have backwards compat.

[Jason2866]

@mhaberler Dont see any issue with S3 and PSRAM. The S3 Tasmota builds (S3 is a mess with OPI...) do compile and work fine with and without PSRAM.
We are using this Arduino 3.0 framework which is compiled with the IDF version used in this setup.
My guess the sdkconfig you use is not 100% correct.

[brentru]

@maxgerhardt Thank you for the initial comment, I didn't know you don't work for PIO Labs.

@Jason2866 Thanks for hosting a mirror of the latest BSP, works perfectly!

[mhaberler]

@Jason2866
I'll give M5GFX a try and then will post a minimal example

[mhaberler]

example ESP-IDF/Arduino project using M5GFX+M5Unified: https://github.com/mhaberler/M5GFX-arduino3

M5GFX fixes: https://github.com/mhaberler/M5GFX/tree/arduino-3.0 - hope these are correct, at least the demo works @lovyan03

@Jason2866 this is needed to get at esp_psram.h: https://github.com/mhaberler/M5GFX-arduino3/blob/main/platformio.ini#L12 - interested if you can reproduce

the stock board file fails to enable PSRAM (missing "memory_type": "qio_qspi")

debugging works fine
interestingly gdb is MUCH faster on a 2015 Intel MBP than on a current M1 Mac; it's not Rosetta, on the M1 gdb is a bona-fine arm64 binary

[maxgerhardt]

I don't see why you'd need framework = arduino, espidf specifically for the project though since nothing fundamental is changed in the sdkconfig / ESP-IDF base. Regular framework = arduino should work too and you can get rid of all CMake files, and possible have working OPI PSRAM out of the box without the -I fix?

[Jason2866]

To have a valid framework with all files at the correct places you have to look at the Arduino Lib Builder how and where the files (especially for the flash types and PSRAM) are copied too. This is totally nonsense at all: -mfix-esp32-psram-cache-issue. Only needed for (a few) ESP32 boards with PSRAM. ALL other ESP32 MCUs don't have this bug.
Using Arduino as a Component of IDF is not always trivial. I do not recommend for Projects just running Arduino Code.