[TW#18940] Debug fails / stops in taskENTER_CRITICAL / queue.c

[nkolban]

I am running a WROVER DevKit with the latest ESP-IDF master as of this date. When I run gdb against my ESP32, all works as desired ... for a period of time. As I step through the statements using n I am presented with the next statement as desired. However, after stepping over a varying number of statements, when I next execute n I do NOT wake up at the next statement, instead I wake up here:

108		ESP_LOGD(LOG_TAG, "Setting up callbacks for logging");
(gdb) n
Target halted. PRO_CPU: PC=0x4008256C (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x4008774D (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x4008774F (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x40087752 (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x40087755 (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x40087758 (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x4008775A (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x4008775C (active)    APP_CPU: PC=0xF8001080 
Target halted. PRO_CPU: PC=0x40083E63 (active)    APP_CPU: PC=0xF8001080 

Program received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 1073412716]
xQueueGenericReceive (xQueue=0x3ffaea14, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0)
    at /home/kolban/particle/Phase2/esp-idf/components/freertos/./queue.c:1455
1455			taskENTER_CRITICAL(&pxQueue->mux);
(gdb) q

This log shows that at line 108 of my source code, I executed n and then wake up at line 1455 of freertos/queue.c:1455. I can not continue from here.

I can recreate this using either command line xtensa-esp32-elf-gdb or Eclipse debugging.

My gdbinit looks as follows:

set remotelogfile gdb_log.txt
target remote :3333
mon reset halt
thb app_main
x $a1=0
c

I can recreate at will using a variety of distinct applications. My platform is Ubuntu 17.10 running under Oracle Virtual Box.

I am attaching both:

• gdb_log.txt
• openocd.log

as requested in the reporting information. I am available to demonstrate live or provide additional documentation on request.

bug.zip

[BobaTrek]

I have experienced the same issue. Please see my submission:
nkolban/esp32-snippets#296

Another programmer also experienced this issue here:
#21

At the end of the above two threads there is a 5 step process that shows how to partially work around the problem of unexpectedly breaking in queue.c, but it does not always work (and is very tedious)

I am really hoping this issue gets figured out!

I am using a USB powered Espressif ESP32-WROVER-KIT V3
On JP8 I originally set 5 jumpers: TMS, TDO, TDI, TCK and S_TDI were closed.
Based on the feedback from another member, I removed the jumper S_TDI and it seems to have made the frequency of occurrence go down.

[worker2345234]

Hello,

same here! I'm using the current IDF from today and the blink example.
$ bin/openocd --version
Open On-Chip Debugger 0.10.0-dev-ga859564 (2017-07-24-16:18)

My jtag adapter configuration: I'm using a USB powered WROVER-Devkit V2.
On JP8 I've set 5 jumpers: TMS, TDO, TDI, TCK and S_TDI are closed.
I left nSRST and GND open.
All other jumpers are open, except JP7 USB 5V.

[foxlav]

Hello,
I am using a USB powered Espressif ESP32-WROVER-KIT V3 and have the same issue.

Alex

[Sermus]

+1
Reproduces for me both with WROVER KIT and in house developed hardware.

[igrr]

@Sermus could you please mention OpenOCD version you are using?

[Sermus]

Sure.
Open On-Chip Debugger 0.10.0-dev-ga859564 (2017-07-25-00:19)

[igrr]

2017-07-25 seems to be pretty dated, could you try the git head?

[BobaTrek]

I am having the same issue with breakpoints.
I just checked my version and it is the same as @worker2345234:
Open On-Chip Debugger 0.10.0-dev-ga859564 (2017-07-24-16:18)

I got this version by following Espressif's esp-idf installation instructions here:
http://esp-idf.readthedocs.io/en/latest/api-guides/jtag-debugging/setup-openocd-windows.html

Espressif's installation instructions say to download the openocd-esp32 version from this link:
https://dl.espressif.com/dl/openocd-esp32-win32-a859564.zip

Checking the GitHub for openocd-esp32 I do see numerous changes since the last time Espressif packaged a binary version for their esp32-idf toolkit and installation instructions.

Does anybody know where we can download a more recently built package of openocd-esp32? I am very hesitant to download the source from GitHub and try to build it myself.

[Sermus]

@igrr Ivan, the OCD's HEAD (0.10.0-dev-g07baed8c) seems to behave better for me.
It doesn't seem to expose the issue initially reported in this thread.

However, it catches sigtrap, please refer to the listing below:

Target halted. PRO_CPU: PC=0x40087D05 (active) APP_CPU: PC=0x400880D9

Program received signal SIGTRAP, Trace/breakpoint trap.
0x40087d05 in is_free (block=0x3ffdc7f4) at esp32/esp-idf/components/heap/./multi_heap.c:123
123 return block->header & BLOCK_FREE_FLAG;
(gdb) c
Continuing.
Target halted. PRO_CPU: PC=0x40087D05 (active) APP_CPU: PC=0x400880D9

Program received signal SIGTRAP, Trace/breakpoint trap.
0x40087d05 in is_free (block=0x3ffdc7f4) at esp32/esp-idf/components/heap/./multi_heap.c:123
123 return block->header & BLOCK_FREE_FLAG;
(gdb) bt
#0 0x40087d05 in is_free (block=0x3ffdc7f4) at esp32/esp-idf/components/heap/./multi_heap.c:123
#1 multi_heap_free_impl (heap=0x3ffc2408, p=0x0) at esp32/esp-idf/components/heap/./multi_heap.c:415
#2 0x40082d6d in heap_caps_free (ptr=0x3ffdc7f8) at esp32/esp-idf/components/heap/./heap_caps.c:275
#3 0x400832f8 in _free_r (r=0x3ffcfba8, ptr=0x3ffdc7f8) at esp32/esp-idf/components/newlib/./syscalls.c:42
#4 0x4000beca in ?? ()
#5 0x400f14c6 in esf_buf_recycle ()
#6 0x400e994e in ppRecycleRxPkt ()
#7 0x400e4730 in ic_ebuf_recycle_rx ()
#8 0x400e167d in sta_input ()
#9 0x400e16d4 in sta_rx_cb ()
#10 0x400ea708 in ppRxPkt ()
#11 0x400ec256 in ppTask ()

[igrr]

When this (sigtrap in multi_heap) happens, do you see any output from the exception handler on the UART console?

[Sermus]

Yes, this is the message:
Guru Meditation Error: Core 1 panic'ed (LoadProhibited)
at pc=4013ff40. Setting bp and returning..

But it never happens if OCD is not run.

[igrr]

It's interesting that the PC reported by the exception handler does not match the backtrace you get from GDB. Can you see what is the output of l *0x4013ff40 in GDB (with the same elf file that produced the exception)?

[igrr]

Also, does this happen when you are running (continue or next) or when you are single stepping (step)?

[Sermus]

Weird things happen. I can't repro sigtrap in the place i reported initially. Now it halts at another location in my user code. It crashes because it tries to dereference nullptr coming as a parameter to a function. The weird is that the caller definitely passes not null as this parameter. I.e. in the function it crashes i see the parameter is null. If i move a frame back down by the stack i see the parameter passed is not null. Stack corruption or something similar happens.

I now reproduced this twice with exactly same symptoms. If i run the app without OCD it executes normally.

I used to use only next and continue while reproducing this.