Error: Core 1 panic'ed (Unhandled debug exception)

[dsyleixa]

Arduino IDE 1.8.9
ESP32 board 1.0.6 (edit; meanwhile updated to 2.0.1)
default settings

generally my program runs fine, but sometimes, unexpectedly, I get this error -
but why and what does that mean....?

Guru Meditation Error: Core 1 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (loopTask)
Core 1 register dump:
PC : 0x400d3254 PS : 0x00060636 A0 : 0x800d3824 A1 : 0x3ffb0030
A2 : 0x0000000e A3 : 0x3ffcc4cc A4 : 0xfffffe77 A5 : 0x00000080
A6 : 0x00000000 A7 : 0x00000001 A8 : 0x3ffc1c24 A9 : 0x00000008
A10 : 0xfffffce1 A11 : 0x00000002 A12 : 0x00000002 A13 : 0x00000353
A14 : 0x0000000a A15 : 0x3ffb08d0 SAR : 0x00000011 EXCCAUSE: 0x00000001
EXCVADDR: 0x00000000 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xfffffff8

ELF file SHA256: 0000000000000000

Backtrace: 0x400d3254:0x3ffb0030 0x400d3821:0x3ffb0170 0x400d3821:0x3ffb02b0 0x400d3821:0x3ffb03f0 0x400d3821:0x3ffb0530 0x400d3821:0x3ffb0670 0x400d3821:0x3ffb07b0 0x400d3821:0x3ffb08f0 0x400d3821:0x3ffb0a30 0x400d3821:0x3ffb0b70 0x400d3821:0x3ffb0cb0 0x400d3821:0x3ffb0df0 0x400d3821:0x3ffb0f30 0x400d3821:0x3ffb1070 0x400d3821:0x3ffb11b0 0x400d3821:0x3ffb12f0 0x400d3821:0x3ffb1430 0x400d3821:0x3ffb1570 0x400d3821:0x3ffb16b0 0x400d3821:0x3ffb17f0 0x400d3821:0x3ffb1930 0x400d3821:0x3ffb1a70 0x400d3821:0x3ffb1bb0 0x400d493c:0x3ffb1cf0 0x400d683b:0x3ffb1e30 0x400d71a4:0x3ffb1ef0 0x400e352d:0x3ffb1fb0 0x4008a1fe:0x3ffb1fd0

Rebooting...

the program is this one:
https://github.com/dsyleixa/Arduino/tree/master/ESP32_GBox/ESP32_Box023
the error happens sometimes when running the "chess" subroutine.

just to mention:
the chess program (i.e., the move generator) is the same as for my Arduino Due and for my Raspberry Pi, and there it works absolutely fine without any problem ever. So IMO the issue here on my ESP32 is probably not related to the chess algorithm itself as far as I can see.

PS, to clraify:
sometimes the move generator crashes already at the 2nd or 5th recursive ply after ~50000 move computations or even less,
sometimes it runs fine through the 7th recursive ply by more than 1 or 2 millions move computations and returns a valid and smart move, e.g.:

2 ply, searched:         9 
 3 ply, searched:       164 
 4 ply, searched:      1018 .
 5 ply, searched:     10045 ..........
 6 ply, searched:    138116 .........................................................................................................
 7 ply, searched:   1099725 
n b8-c6

0 ply, searched:         1 
 1 ply, searched:         2 
 2 ply, searched:        29 
 3 ply, searched:       466 
 4 ply, searched:      3262 ..
 5 ply, searched:     32422 ................
 6 ply, searched:    251225 
B c1-g5  

0 ply, searched:         1 
 1 ply, searched:         2 
 2 ply, searched:        63 
 3 ply, searched:       741 
 4 ply, searched:      2559 ....
 5 ply, searched:     31058 ..Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (loopTask) 

[dsyleixa]

update:
different game by different moves, but similar error after a while - still no clue what's happening here:

Guru Meditation Error: Core 1 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (loopTask)
Core 1 register dump:
PC : 0x400d3254 PS : 0x00060836 A0 : 0x800d3824 A1 : 0x3ffb0030
A2 : 0x00000007 A3 : 0x3ffcc4cc A4 : 0xfffffcec A5 : 0x00000080
A6 : 0x00000000 A7 : 0x00000001 A8 : 0x3ffc1c8f A9 : 0x00000008
A10 : 0x00000008 A11 : 0x00000001 A12 : 0x00000005 A13 : 0x00000020
A14 : 0x00000020 A15 : 0x3ffb08d0 SAR : 0x00000011 EXCCAUSE: 0x00000001
EXCVADDR: 0x00000000 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xfffffff8

ELF file SHA256: 0000000000000000

Backtrace: 0x400d3254:0x3ffb0030 0x400d3821:0x3ffb0170 0x400d3821:0x3ffb02b0 0x400d3821:0x3ffb03f0 0x400d3821:0x3ffb0530 0x400d3821:0x3ffb0670 0x400d3821:0x3ffb07b0 0x400d3821:0x3ffb08f0 0x400d3821:0x3ffb0a30 0x400d3821:0x3ffb0b70 0x400d3821:0x3ffb0cb0 0x400d3821:0x3ffb0df0 0x400d3821:0x3ffb0f30 0x400d3821:0x3ffb1070 0x400d3821:0x3ffb11b0 0x400d3821:0x3ffb12f0 0x400d3821:0x3ffb1430 0x400d3821:0x3ffb1570 0x400d3821:0x3ffb16b0 0x400d3821:0x3ffb17f0 0x400d3821:0x3ffb1930 0x400d3821:0x3ffb1a70 0x400d3821:0x3ffb1bb0 0x400d4968:0x3ffb1cf0 0x400d6867:0x3ffb1e30 0x400d71d0:0x3ffb1ef0 0x400e3559:0x3ffb1fb0 0x4008a1fe:0x3ffb1fd0

Rebooting...
ets Jun 8 2016 00:22:57

[SuGlider]

@dsyleixa
Debug exception reason: Stack canary watchpoint triggered (loopTask)
This message points to stack related issues in loop() or in any other function that is called from it.

Some nice explanation about possible reasons for this error can be found at this link:
https://arduino.stackexchange.com/questions/80729/esp32-stack-canary-watchpoint-triggered

Good Luck!

[SuGlider]

This tool will help you in debugging this issue:
https://github.com/me-no-dev/EspExceptionDecoder

You can decode the backtrace message and find out where the exception was thrown.

[dsyleixa]

I did this, but I don't understand what to c+p into the field and how to proceed
if I paste the Backtrace

0x400d3254:0x3ffb0030 0x400d3821:0x3ffb0170 0x400d3821:0x3ffb02b0 0x400d3821:0x3ffb03f0 0x400d3821:0x3ffb0530 0x400d3821:0x3ffb0670 0x400d3821:0x3ffb07b0 0x400d3821:0x3ffb08f0 0x400d3821:0x3ffb0a30 0x400d3821:0x3ffb0b70 0x400d3821:0x3ffb0cb0 0x400d3821:0x3ffb0df0 0x400d3821:0x3ffb0f30 0x400d3821:0x3ffb1070 0x400d3821:0x3ffb11b0 0x400d3821:0x3ffb12f0 0x400d3821:0x3ffb1430 0x400d3821:0x3ffb1570 0x400d3821:0x3ffb16b0 0x400d3821:0x3ffb17f0 0x400d3821:0x3ffb1930 0x400d3821:0x3ffb1a70 0x400d3821:0x3ffb1bb0 0x400d4968:0x3ffb1cf0 0x400d6867:0x3ffb1e30 0x400d71d0:0x3ffb1ef0 0x400e3559:0x3ffb1fb0 0x4008a1fe:0x3ffb1fd0

then nothing happens.
no button to press and no action follows

[dsyleixa]

PS, just to mention,
I also had added lots of delay(1) in between my for() and while() loops not to block the scheduler.
I also added Serial.print('.') debug outputs to indicate incidental blocking,
but when the Panic Error happens then it's in a <500ms interval since the last Serial.print('.')
so it clearly does not block the scheduler watchdog.

[atanisoft]

@dsyleixa This has nothing to do with the task scheduler, this is entirely on using more than 8kb of stack space in setup()/loop(). If you have large arrays (such as: uint8_t something[5000]) it may crash upon entry to the method, this is the most common reason for stack canary crashes like you post.

[dsyleixa]

Hmmmh...
let me elaborate on it...:

Apart from Chess(), the entire program runs fine on the ESP32 through setup() and loop() and also may call other subprograms such as Paint() or Pong() without any issues.
The error never occurs with different sub-programs, just with the subprogram "chess".
All subprograms are using the same global variables which are used by setup() and loop().

So all runs fine >>>>>>>> untill Chess() is run.

But also the first couples of chess moves are always fine though, so also Chess() does not violate the RAM size limit when starting.
Also manual moves are apllied correctly, calling the auto move generator just for move legality checks, and then also the first couples of auto moves, too.

Furthermore for Chess(), the error happens not always and not reproducably,
sometimes after the 7th auto move generation ply, sometimes even after the 3rd, or perhaps after the 8th or 9th,
always by identical boardsettings and identical move series.
That actually makes me doubt that it's a RAM size issue.

OTOH, I meanwhile tested the Chess subprogram also on my Mega2560 too (because of smaller RAM than on Due or Raspi) , and also over there it always runs fine,
so IMO it probably cannot happen because of the Chess() variables on the stack (CMIIW).

when compiling, the IDE says:

The sketch uses 854266 bytes (65%) of the program memory. The maximum is 1310720 bytes.
Global variables use 48692 bytes (14%) of dynamic memory, leaving 278988 bytes for local variables. The maximum is 327680 bytes. 

I am completely at a loss, tbh...

[atanisoft]

IDE output of memory usage is not applicable to task stack sizing. It is only applicable to global variable allocations (one that you don't create via new/malloc/etc) and for overall size of the program with respect to the partition size.

Comparing ESP32 to an AVR Mega2650 is not a good argument for "it works", they are entirely different architectures AND the Mega2650 does not use task stacks but instead allocates on heap directly which is not applicable in an RTOS environment.

Since you have not shared much in the way of code nobody will be able to point out where your program is going awry other than general ideas like @SuGlider and I've posted.

[dsyleixa]

I actually already shared the code above, in the TOP:
https://github.com/dsyleixa/Arduino/tree/master/ESP32_GBox/ESP32_Box023

[SuGlider]

My general guess is about Stack Overwflow because of potential Chess recursion.

The main difference from ESP32 Arduino to other Chips Arduino is that in ESP32 everything is running under FreeRTOS, thus, as @atanisoft said, loop() and setup() are tasks with a limit of 8K Stack each. You can create a separated task to specific routines (such as Chess, with higher Stack size for that task).

For the other Chips, Arduino is built as a pure Bare Metal application and Stack can possibly reach higher limits in available RAM, depending on the way it was built and configured. So it could explain why you don't see any errors with other "Chip Arduinos".

From the link I posted there is a general explanation:
https://arduino.stackexchange.com/questions/80729/esp32-stack-canary-watchpoint-triggered

• recursive functions - each time a function recurses it uses stack space. If it recurses deeply enough then it will trample the stack guard and cause this exception. For instance:

int count(i) {
  i--;

  if(i > 0) {
    Serial.println(count(i));
  }

  return i;
}

void loop() {
  count(8000);
}

Each time a function recurses, its return address and its arguments and local variables are all stored on the stack. If it recurses too many times it will use more storage than is allocated to the stack.

[dsyleixa]

well, as already stated, if it was a RAM issue then it's supposed to happen always reproducably at the same time, but it does not!
See here:
#6010 (comment)

sometimes the move generator crashes already at the 2nd or 5th recursive ply after ~50000 move computations or even less,
sometimes it runs fine through the 7th recursive ply by more than 1 or 2 millions move computations and returns a valid and smart move, e.g.:

2 ply, searched:         9 
 3 ply, searched:       164 
 4 ply, searched:      1018 .
 5 ply, searched:     10045 ..........
 6 ply, searched:    138116 .........................................................................................................
 7 ply, searched:   1099725 
n b8-c6

0 ply, searched:         1 
 1 ply, searched:         2 
 2 ply, searched:        29 
 3 ply, searched:       466 
 4 ply, searched:      3262 ..
 5 ply, searched:     32422 ................
 6 ply, searched:    251225 
B c1-g5  

0 ply, searched:         1 
 1 ply, searched:         2 
 2 ply, searched:        63 
 3 ply, searched:       741 
 4 ply, searched:      2559 ....
 5 ply, searched:     31058 ..Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (loopTask) 

[dsyleixa]

update:
I meanwhile even decreased the max deepening and the HashTable size and it sometimes already crashes in the 4th deepening ply whilst earlier it had successfully calculated up to the 7th ply.
So IMO it really can't be a recursive RAM capture thing actually.

  A B C D E F G H 
  --------------- 
8 r . . q k b . r 8 
7 + + + b n + + + 7 
6 . . . . + . . . 6 
5 . . . + . . . . 5 
4 . n . * . . . . 4 
3 . . N . * N . . 3 
2 * * * . B * * * 2 
1 R . B Q K . . R 1 
  --------------- 
  A B C D E F G H 
> WHITE: 
 DEBUG cstring : 
 DEBUG K: 8000  
 DEBUG L: 19 

 0 ply, searched:         1 
 1 ply, searched:         2 
 2 ply, searched:       193 
 3 ply, searched:       542 
 4 ply, searched:      5316 ..........Guru Meditation Error: Core  1 panic'ed (Unhandled debug exception)
Debug exception reason: Stack canary watchpoint triggered (loopTask) 
Core 1 register dump:
PC      : 0x400d3310  PS      : 0x00060036  A0      : 0x800d38da  A1      : 0x3ffb0010  
A2      : 0xffffffe9  A3      : 0x3ffc1d58  A4      : 0x3ffbdc08  A5      : 0x00000080  
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x3ffc1bdc  A9      : 0x00000008  
A10     : 0x00000008  A11     : 0x00000001  A12     : 0x00000005  A13     : 0x00000020  
A14     : 0x00000020  A15     : 0x3ffb08b0  SAR     : 0x00000011  EXCCAUSE: 0x00000001  
EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffff8  

ELF file SHA256: 0000000000000000

Backtrace: 0x400d3310:0x3ffb0010 0x400d38d7:0x3ffb0150 0x400d38d7:0x3ffb0290 0x400d38d7:0x3ffb03d0 0x400d38d7:0x3ffb0510 0x400d38d7:0x3ffb0650 0x400d38d7:0x3ffb0790 0x400d38d7:0x3ffb08d0 0x400d38d7:0x3ffb0a10 0x400d38d7:0x3ffb0b50 0x400d38d7:0x3ffb0c90 0x400d38d7:0x3ffb0dd0 0x400d38d7:0x3ffb0f10 0x400d38d7:0x3ffb1050 0x400d38d7:0x3ffb1190 0x400d38d7:0x3ffb12d0 0x400d38d7:0x3ffb1410 0x400d38d7:0x3ffb1550 0x400d38d7:0x3ffb1690 0x400d38d7:0x3ffb17d0 0x400d38d7:0x3ffb1910 0x400d38d7:0x3ffb1a50 0x400d38d7:0x3ffb1b90 0x400d4a65:0x3ffb1cd0 0x400d6993:0x3ffb1e30 0x400d72f4:0x3ffb1ef0 0x400e365d:0x3ffb1fb0 0x4008a1fe:0x3ffb1fd0

Rebooting...

Nonetheless, after pasting the Backtrace into the Exception decoder then still nothing happens at all...

you may check a downstripped standallone version here (no TFT hardware etc):
https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino

BTW, is it possible to disable this eff*** "Guru"...?

[atanisoft]

Suffice to say, recursion within tasks is not an easy problem to solve. It's not really a task that is designed to run on an embedded RTOS platform entirely.

However, as noted previously, you can create a task with a larger stack size to run your recursion process.

BTW, is it possible to disable this eff*** "Guru"...?

This is coming from the pre-built ESP-IDF code with the default setting of CONFIG_ESP_SYSTEM_PANIC_PRINT_REBOOT. You would need to rebuild ESP-IDF code with CONFIG_ESP_SYSTEM_PANIC_SILENT_REBOOT to not have it print the register dump and backtrace, though it may print other details.

[dsyleixa]

I do not run a recursion in a task.
Neither in my GBox program nor in the downstripped demo version
https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino

But that eff*** Guru error happens at either program - nonetheless, never any runtime errors e.g. on my MEGA or my DUE.

[atanisoft]

I do not run a recursion in a task.

Both setup() and loop() run in a single RTOS task with 8kb stack.

https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino#L9 describes using recursion as part of it's algorithm.

Recursion points:

• Initial entry point: https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino#L462 (calls https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino#L107)
• First point of Recursion: https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino#L161
• Second point of Recursion: https://github.com/dsyleixa/Arduino/blob/master/Chess/chess0048e32/chess0048e32.ino#L224

Each level of recursion depth will use at least 175b of stack plus any additional required for making function calls. At some point in the recursion depth it will fail as you have found.

neverany errors e.g. on my MEGA or my DUE.

AVR doesn't use RTOS and doesn't have the same concept of task stack. It uses all free heap/SRAM for the recursion usage, very likely at a certain depth of recursion it will start randomly overwriting areas of SRAM or perhaps simply crash.