Closed
Description
IDE / Tooling
PlatformIO
What happened?
I am running ESP-DASH-Pro 4.6.2 on an ESP32 using ESPAsyncWebServer 3.7.7 and AsyncTCP 3.4.0. My application requires that the ESP32 does not reset itself because it is connected to a relay holding the power up of the ESP32. So as soon as the ESP resets, the relay gets released and the ESP32 dies due to power down.
That said, I tried to figure out why the ESP32 dies from time to time (currently approx. once every 1-3 days). To do that I changed the power supply to keep it running even after a reset and added some code to write the coredumps to LittleFS to be able to check them regularly. I am already doing this for months now, but I was not able to catch a good coredump leading me closer to the issue.
As far as I got, it seems like there is a pointer of a callback which points to some very strange memory region (0x8018d038) which gets executed resulting in an InstFetchPorohibited Hardfault. As the thread with these strange pcb is named asyncTCP it seems like its happening somewhere in the AsyncTCP code. Unfortunately the stack trace is now showing the last moment before the strange memory region get called. So I am clueless where to start looking, as it could be a memory leak anywhere in the codebase, or the pointer gets called after it was released already.
I now updated AsyncTCP again to the new version and wait for the next reset to happen.
Any help is appreciated.
Stack Trace
===============================================================
==================== ESP32 CORE DUMP START ====================
Crashed task handle: 0x3ffe05ec, name: '', GDB name: 'process 1073612268'
Crashed task is not in the interrupt context
================== CURRENT THREAD REGISTERS ===================
exccause 0x14 (InstFetchProhibitedCause)
excvaddr 0x8018d038
epc1 0x40084d77
epc2 0x0
epc3 0x0
epc4 0x0
epc5 0x40082553
epc6 0x0
eps2 0x0
eps3 0x0
eps4 0x0
eps5 0x60a20
eps6 0x0
[New process 1073612268]
[New process 1073431312]
[New process 1073467184]
[New process 1073468712]
[New process 1073447304]
[New process 1073469084]
[New process 1073443912]
[New process 1073445456]
[New process 1073437912]
[New process 1073528948]
[New process 1073437540]
[New process 1073464584]
[New process 1073439140]
[Current thread is 1 (process 1073612268)]
==================== CURRENT THREAD STACK =====================
======================== THREADS INFO =========================
pc 0x8018d039 0x8018d039
lbeg 0x4008a448 1074308168
lend 0x4008a45e 1074308190
lcount 0xffffffff 4294967295
sar 0x14 20
ps 0x60920 395552
threadptr <unavailable>
br <unavailable>
scompare1 <unavailable>
acclo <unavailable>
acchi <unavailable>
m0 <unavailable>
m1 <unavailable>
m2 <unavailable>
m3 <unavailable>
expstate <unavailable>
f64r_lo <unavailable>
f64r_hi <unavailable>
f64s <unavailable>
fcr <unavailable>
fsr <unavailable>
a0 0x8018d500 -2145856256
a1 0x3ffdf990 1073609104
a2 0x312e3104 825110788
a3 0xfc 252
a4 0x3ffe0a54 1073613396
a5 0x31aa00 3254784
a6 0x3ffe20b8 1073619128
a7 0xf7e5d5e0 -135932448
a8 0x800e0074 -2146566028
a9 0x3ffdb9ac 1073592748
a10 0x3ffc7d08 1073511688
a11 0x1ffefcce 536804558
a12 0x3ffdf998 1073609112
a13 0x60123 393507
a14 0xb33fffff -1287651329
a15 0xcdcd 52685
Retrying reading threads information...
Retrying reading threads information...
TCB NAME PRIO C/B STACK USED/FREE
---------- ---------------- -------- ----------------
0x3ffe05ec Corrupted TCB data
0x3ffb4310 async_tcp1073430480/1073426444 178044/324
0x3ffbcf30 1073466688/1073466028 1073467218/132
0x3ffbd528 1073468208/1073467556 1073468764/140
0x3ffb8188 1073421648/1073414944 1073447356/24236
0x3ffbd69c 1073550944/1073547264 1073469135/82208
0x3ffb7448 1073443328/1073442756 1073443963/188
0x3ffb7a50 1073444864/1073444300 1073445484/196
0x3ffb5cd8 1073411520/1073408156 1073437940/25724
0x3ffcc074 1073585392/1073581796 1073528981/56888
0x3ffb5b64 1073436704/1073432672 1073437591/328
0x3ffbc508 1073463616/1073455364 1073464616/572
0x3ffb61a4 1073558736/10735525961073439170/120056
==================== THREAD 1 (TCB: 0x3ffe05ec, name: '') =====================
==================== THREAD 2 (TCB: 0x3ffb4310, name: 'async_tcp') =====================
#0 0x8018d039 in ?? ()
#1 0x8018d500 in ?? ()
#2 0x8018d78d in ?? ()
#3 0x8018d7d5 in ?? ()
#4 0x8008fb81 in ?? ()
==================== THREAD 3 (TCB: 0x3ffbcf30, name: '') =====================
#0 0x4010750e in tcp_receive (pcb=0x3ffb4010) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c:1264
==================== THREAD 4 (TCB: 0x3ffbd528, name: '') =====================
#0 0x4008d45a in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1 0x400f40ec in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2 0x40090de6 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3 0x4008fb81 in vPortTaskWrapper (pxCode=0x40090d6c <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 5 (TCB: 0x3ffb8188, name: '') =====================
#0 0x4008d45a in esp_cpu_wait_for_intr () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/cpu.c:64
#1 0x400f40ec in esp_vApplicationIdleHook () at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:58
#2 0x40090de6 in prvIdleTask (pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:4353
#3 0x4008fb81 in vPortTaskWrapper (pxCode=0x40090d6c <prvIdleTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 6 (TCB: 0x3ffbd69c, name: '') =====================
#0 0x400835e0 in esp_crosscore_int_send_yield (core_id=1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/crosscore_int.c:121
#1 0x40091cf1 in vTaskDelay (xTicksToDelay=10) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/xtensa/include/xt_utils.h:41
#2 0x400ecf3c in delay (ms=10) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-misc.c:210
#3 0x4019464c in HTTPClient::handleHeaderResponse (this=0x3ffb204c) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:1288
#4 0x401948b9 in HTTPClient::sendRequest (this=0x3ffb204c, type=<optimized out>, payload=<optimized out>, size=<optimized out>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:617
#5 0x401949ba in HTTPClient::GET (this=0x3ffb204c) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/HTTPClient/src/HTTPClient.cpp:500
#6 0x400e0be2 in EasyDDNSClass::update (this=0x3ffc7e6c <EasyDDNS>, ddns_update_interval=10000, use_local_ip=<optimized out>) at .pio/libdeps/henry-central-control/EasyDDNS/EasyDDNS.cpp:40
#7 0x400d5d30 in loop () at src/main.cpp:823
#8 0x400ef3eb in loopTask (pvParameters=0x0) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:74
#9 0x4008fb81 in vPortTaskWrapper (pxCode=0x400ef3b0 <loopTask(void*)>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 7 (TCB: 0x3ffb7448, name: '') =====================
#0 0x400835e0 in esp_crosscore_int_send_yield (core_id=1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/crosscore_int.c:121
#1 0x40090754 in prvProcessTimerOrBlockTask (xNextExpireTime=0, xListWasEmpty=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/xtensa/include/xt_utils.h:41
#2 prvTimerTask (pvParameters=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/timers.c:685
#3 0x4008fb81 in vPortTaskWrapper (pxCode=0x40090644 <prvTimerTask>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 8 (TCB: 0x3ffb7a50, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4 0x40082749 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5 0x4008fb81 in vPortTaskWrapper (pxCode=0x40082708 <ipc_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 9 (TCB: 0x3ffb5cd8, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4 0x40082749 in ipc_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_ipc.c:62
#5 0x4008fb81 in vPortTaskWrapper (pxCode=0x40082708 <ipc_task>, pvParameters=0x1) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 10 (TCB: 0x3ffcc074, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffb44d8) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x4008f833 in xQueueReceive (xQueue=0x3ffb4484, pvBuffer=0x3ffaf70c, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4 0x40189d48 in NetworkEvents::_checkForEvent (this=<optimized out>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:111
#5 NetworkEvents::_checkForEvent (this=0x3ffcbda4 <Network+4>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:100
#6 0x40189e18 in operator() (__closure=0x0, self=0x3ffcbda4 <Network+4>) at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:66
#7 _FUN () at C:/Users/MichaelHummels/.platformio/packages/framework-arduinoespressif32/libraries/Network/src/NetworkEvents.cpp:67
#8 0x4008fb81 in vPortTaskWrapper (pxCode=0x40189e10 <_FUN(void*)>, pvParameters=0x3ffcbda4 <Network+4>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 11 (TCB: 0x3ffb5b64, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffd8b5c) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x4008f833 in xQueueReceive (xQueue=0x3ffd8b08, pvBuffer=0x3ffd9e38, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4 0x4019d218 in _mdns_service_task (pvParameters=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/managed_components/espressif__mdns/mdns.c:5417
#5 0x4008fb81 in vPortTaskWrapper (pxCode=0x4019d1f4 <_mdns_service_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 12 (TCB: 0x3ffbc508, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffb45c4) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x4008f833 in xQueueReceive (xQueue=0x3ffb4570, pvBuffer=0x3ffb5970, xTicksToWait=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1632
#4 0x401bd708 in esp_event_loop_run (event_loop=0x3ffb2c54, ticks_to_run=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:644
#5 0x401bd8c2 in esp_event_loop_run_task (args=0x3ffb2c54) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_event/esp_event.c:106
#6 0x4008fb81 in vPortTaskWrapper (pxCode=0x401bd8b8 <esp_event_loop_run_task>, pvParameters=0x3ffb2c54) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
==================== THREAD 13 (TCB: 0x3ffb61a4, name: '') =====================
#0 0x4000bff0 in ?? ()
#1 0x4008fdf1 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include/freertos/portmacro.h:560
#2 vPortExitCritical (mux=0x3ffbdfcc <xKernelLock>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:514
#3 0x40092251 in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=4294967295) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/tasks.c:5765
#4 0x40137424 in timer_task (arg=<optimized out>) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_timer/src/esp_timer.c:459
#5 0x4008fb81 in vPortTaskWrapper (pxCode=0x40137414 <timer_task>, pvParameters=0x0) at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:139
======================= ALL MEMORY REGIONS ========================
Name Address Size Attrs
.rtc.text 0x400c0000 0x0 RW
.rtc.dummy 0x3ff80000 0x0 RW
.rtc.force_fast 0x3ff80000 0x1c RW A
.rtc_noinit 0x50000200 0x0 RW
.rtc.force_slow 0x50000200 0x0 RW
.iram0.vectors 0x40080000 0x403 R XA
.iram0.text 0x40080404 0x16c83 R XA
.dram0.data 0x3ffbdb60 0x676c RW A
.ext_ram_noinit 0x3f800000 0x0 RW
.ext_ram.bss 0x3f800000 0x0 RW
.flash.appdesc 0x3f400020 0x100 R A
.flash.rodata 0x3f400120 0x6a1e0 RW A
.flash.text 0x400d0020 0xf02c4 R XA
.iram0.data 0x40097088 0x0 RW
.iram0.bss 0x40097088 0x0 RW
.dram0.heap_start 0x3ffcf188 0x0 RW
.coredump.tasks.data 0x3ffb4310 0x160 RW
.coredump.tasks.data 0x3ffb3fd0 0x230 RW
.coredump.tasks.data 0x3ffe05ec 0x160 RW
.coredump.tasks.data 0x3ffdf8d0 0x1f0 RW
.coredump.tasks.data 0x3ffbcf30 0x160 RW
.coredump.tasks.data 0x3ffbcd40 0x1a0 RW
.coredump.tasks.data 0x3ffbd528 0x160 RW
.coredump.tasks.data 0x3ffbd330 0x1a0 RW
.coredump.tasks.data 0x3ffb8188 0x160 RW
.coredump.tasks.data 0x3ffb1d50 0x5c0 RW
.coredump.tasks.data 0x3ffbd69c 0x160 RW
.coredump.tasks.data 0x3ffd1660 0x190 RW
.coredump.tasks.data 0x3ffb7448 0x160 RW
.coredump.tasks.data 0x3ffb7200 0x1c0 RW
.coredump.tasks.data 0x3ffb7a50 0x160 RW
.coredump.tasks.data 0x3ffb7800 0x1c0 RW
.coredump.tasks.data 0x3ffb5cd8 0x160 RW
.coredump.tasks.data 0x3ffaf5c0 0x2d0 RW
.coredump.tasks.data 0x3ffcc074 0x160 RW
.coredump.tasks.data 0x3ffd9cf0 0x1f0 RW
.coredump.tasks.data 0x3ffb5b64 0x160 RW
.coredump.tasks.data 0x3ffb5820 0x230 RW
.coredump.tasks.data 0x3ffbc508 0x160 RW
.coredump.tasks.data 0x3ffbc140 0x1c0 RW
.coredump.tasks.data 0x3ffb61a4 0x160 RW
.coredump.tasks.data 0x3ffd34d0 0x200 RW
===================== ESP32 CORE DUMP END =====================
===============================================================
Done!
Minimal Reproductible Example (MRE)
This is a hard one, as this is a very sporadic problem and only occurs once in a few days.
I confirm that:
- I have read the documentation.
- I have searched for similar discussions.
- I have searched for similar issues.
- I have looked at the examples.
- I have upgraded to the lasted version of AsyncTCP.
Metadata
Metadata
Assignees
Labels
No labels