[TW#17765] I2C crashing - watchdog timeout (master & v3.0 branch)

[DavidAntliff]

I have a project that has three I2C slave devices on a single bus (running at 100kHz). For some time I was developing with ESP-IDF 2.1.1 and everything was working pretty well, except for a weird problem where the I2C master would freeze up after a few minutes. I did some research and it looks like this is a problem with the I2C master hardware state machine which has been addressed in more recent commits of ESP-IDF. So to make use of this fix I migrated my project to use master (595688a). I had to make a few changes (remove references to FreeRTOS heap measurement commands, add nvs_flash_init() before initialising WiFi) but then everything seemed to work well. The slave devices are all being polled correctly and everything seems happy.

The project is here: https://github.com/DavidAntliff/esp32-poolmon/tree/ESP-IDF_master

I came back a little while later and the application is crashing over and over with the following console output shortly after boot:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
Register dump:
PC      : 0x400859f3  PS      : 0x00060034  A0      : 0x80084685  A1      : 0x3ffb0590  
0x400859f3: xQueueGenericSendFromISR at /Users/david/esp32/esp-idf-master/components/freertos/./queue.c:2037

A2      : 0x00000001  A3      : 0x00000000  A4      : 0x3ffb05b0  A5      : 0x00000002  
A6      : 0x3ffbc970  A7      : 0x00060021  A8      : 0x800859f3  A9      : 0x3ffb0570  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x00000002  A13     : 0x3ffbadc0  
A14     : 0x00000000  A15     : 0x400849fc  SAR     : 0x00000012  EXCCAUSE: 0x00000005  
0x400849fc: i2c_isr_handler_default at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  

Backtrace: 0x400859f3:0x3ffb0590 0x40084682:0x3ffb05b0 0x40084a89:0x3ffb05e0 0x40082ba5:0x3ffb0610 0x4000bfed:0x00000000
0x400859f3: xQueueGenericSendFromISR at /Users/david/esp32/esp-idf-master/components/freertos/./queue.c:2037

0x40084682: i2c_master_cmd_begin_static at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

0x40084a89: i2c_isr_handler_default at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

0x40082ba5: _xt_lowint1 at /Users/david/esp32/esp-idf-master/components/freertos/./xtensa_vectors.S:1105

Rebooting...

A software or on-board reset does not stop this endless reset behaviour, however removing power for a short period of time does "fix" the issue. It is strange that a brief ESP32 reset does not clear it. (EDIT: but a long reset press does).

[DavidAntliff]

In addition, I've not seen the problem occur with WiFi disabled, but when I enable WiFi I've now seen it happen about five times.

[DavidAntliff]

Another dump with slightly different info:

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
Register dump:
PC      : 0x4008550d  PS      : 0x00060034  A0      : 0x800859a0  A1      : 0x3ffb0570  
0x4008550d: prvCopyDataToQueue at /Users/david/esp32/esp-idf-master/components/freertos/./queue.c:2037

A2      : 0x3ffbc928  A3      : 0x00000000  A4      : 0x00000002  A5      : 0x3ffbadc0  
A6      : 0x00000000  A7      : 0x400849fc  A8      : 0x3ffbc97c  A9      : 0x3ffbc978  
0x400849fc: i2c_isr_handler_default at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

A10     : 0x3ffbc978  A11     : 0x3ffb05b8  A12     : 0x00000004  A13     : 0x3ffbc97c  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000020  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  

Backtrace: 0x4008550d:0x3ffb0570 0x4008599d:0x3ffb0590 0x40084682:0x3ffb05b0 0x40084a89:0x3ffb05e0 0x40082ba5:0x3ffb0610 0x4000bfed:0x00000000
0x4008550d: prvCopyDataToQueue at /Users/david/esp32/esp-idf-master/components/freertos/./queue.c:2037

0x4008599d: xQueueGenericSendFromISR at /Users/david/esp32/esp-idf-master/components/freertos/./queue.c:2037

0x40084682: i2c_master_cmd_begin_static at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

0x40084a89: i2c_isr_handler_default at /Users/david/esp32/esp-idf-master/components/driver/./i2c.c:1023

0x40082ba5: _xt_lowint1 at /Users/david/esp32/esp-idf-master/components/freertos/./xtensa_vectors.S:1105


Rebooting...

[DavidAntliff]

Also, if I hold down the reset button on my ESP32 board for about a second (rather than just pressing it quickly) then the issue does clear on next boot. Pressing it too quickly doesn't help.

[DavidAntliff]

From what I can tell so far, this only occurs when there's a glitch on the I2C bus and WiFi is enabled. If WiFi is disabled, the glitch may happen but it doesn't cause this cascade-reset issue.

[DavidAntliff]

I've now seen this happen with the v3.0 branch (e7dc749).

[DavidAntliff]

To be clear, I am using a fairly long cable with my I2C light sensor (TSL2561) - this cable has the side effect of causing the occasional I2C error (missing slave ACK or timeout). Normally this doesn't cause a problem (apart from the failed transaction) however it appears to trigger this issue occasionally. If I use a much shorter cable, the occasional I2C error does not occur, and neither does this issue. So the cable is clearly related to the issue, but I believe it is triggering the problem by inducing I2C errors, rather than causing it. I think the I2C master shouldn't fail like this in the event of a slave or comms issue.

[DavidAntliff]

I have now observed this issue occur when WiFi is disabled.

[costaud]

Thanks for the detailed descriptions.

I have a question that how large is the pull-up resister?

We will have some tests under similar condition and get back to you.

[panfeng-espressif]

Thanks for the detailed descriptions.
I would like to ask what is esp_mqtt.h? I could not find it in either esp-idf3.0 or esp-idf2.1.
If you could share it, that will help us debug this issue.

[negativekelvin]

@panfeng-espressif https://github.com/DavidAntliff/esp-mqtt/

It is a submodule of his other project

[DavidAntliff]

SDA and SCK pull up resistance measured (with ESP32, and AVR, removed) at 7.4 kOhm each.

Not sure if it's relevant but I sometimes get a similar error (and resulting continual crash) involving RMT (this time with ESP IDF v3.0):

Guru Meditation Error: Core  0 panic'ed (LoadProhibited)
. Exception was unhandled.
Register dump:
PC      : 0x40084e99  PS      : 0x00060033  A0      : 0x80081aa4  A1      : 0x3ffb05c0  
0x40084e99: rmt_driver_isr_default at /Users/david/esp32/esp-idf-v3.0/components/driver/./rmt.c:497

A2      : 0x00000006  A3      : 0x00000040  A4      : 0x00000002  A5      : 0x00000000  
A6      : 0x00000000  A7      : 0x00001800  A8      : 0x00000006  A9      : 0x00000001  
A10     : 0x00000001  A11     : 0x3ffb05c0  A12     : 0x80082d8b  A13     : 0x00000001  
A14     : 0x00060021  A15     : 0x00020d90  SAR     : 0x00000014  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000014  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0x00000000  

Backtrace: 0x40084e99:0x3ffb05c0 0x40081aa1:0x3ffb05f0 0x40082b25:0x3ffb0610 0x4000bfed:0x00000000
0x40084e99: rmt_driver_isr_default at /Users/david/esp32/esp-idf-v3.0/components/driver/./rmt.c:497

0x40081aa1: shared_intr_isr at /Users/david/esp32/esp-idf-v3.0/components/esp32/./intr_alloc.c:754

0x40082b25: _xt_lowint1 at /Users/david/esp32/esp-idf-v3.0/components/freertos/./xtensa_vectors.S:1105


Rebooting...

Might be a red herring though...

[DavidAntliff]

On several occasions the system has run for 3+ days without me seeing this error. Other times it can happen after just a few minutes since power up. I have noticed that it doesn't seem to crash, or enter this continual crash loop, if I don't have a serial console connected, but I need to do more tests to confirm this for sure.

[koobest]

Hi @DavidAntliff ,
Thanks for you feedback. we are now want to reproducing your problem. is there any place in your code that I should to modify ?