Cypress: FPGA: UART random failure

[yarbcy]

Description of defect

Cypress: FPGA: UART random failure

Step to reproduce:

1. Run UART tests
2. Comment all test in mbed_hal_fpga_ci_test_shield/uart/main.cpp, uncomment any test for example line 313.
3. Comment line 252 - 271
4. Run test (repeate couple times)

Expected: PASS

Actual:
tester.reset() returned:

++ MbedOS Error Info ++
[1573648317.62][CONN][RXD] Error Status: 0x80010133 Code: 307 Module: 1
[1573648317.68][CONN][RXD] Error Message: Mutex: 0x8003B14, Not allowed in ISR context
[1573648317.70][CONN][RXD] Location: 0x100095AD
[1573648317.73][CONN][RXD] Error Value: 0x8003B14
[1573648317.84][CONN][RXD] Current Thread: main Id: 0x8003B38 Entry: 0x10009477 StackSize: 0x1000 StackMem: 0x8004A18 SP: 0x80055D4
[1573648317.94][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80010133&tgt=CY8CKIT_062_WIFI_BT

This test PASSED if comment line 251.

Target(s) affected by this defect ?

All Cypress boards. Tested on CY8CKIT_062_WIFI_BT

Toolchain(s) (name and version) displaying this defect ?

Tested on GCC_ARM

What version of Mbed-os are you using (tag or sha) ?

Latest from main.
f1848f9

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

N/A

How is this defect reproduced ?

50/50

[yarbcy]

@0xc0170 @mprse

[0xc0170]

Latest from main.

please report SHA, helps to get fixed version, edit the above to contain SHA.

[yarbcy]

Latest from main.

please report SHA, helps to get fixed version, edit the above to contain SHA.

Done. Thanks.

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2315

[yarbcy]

@mprse Please take a llok if you have time.

[yarbcy]

@mprse I see this exception from tester (FPGA) side.

[mprse]

So if you run only one test case (any) it fails randomly?

Comment line 252 - 271

What does it mean? What is the purpose of removing these lines:

mbed-os/TESTS/mbed_hal_fpga_ci_test_shield/uart/main.cpp

Lines 252 to 271 in 6993724

	core_util_critical_section_exit();
	tester.rx_start();
	tester.tx_start(use_flow_control);
	while (core_util_atomic_load_u32(&td.rx_cnt) != GETC_REPS) {
	// Wait until the last byte is received to UART RX reg.
	};
	core_util_critical_section_enter();
	serial_irq_set(&serial, RxIrq, 0);
	core_util_critical_section_exit();
	tester.tx_stop();
	tester.rx_stop();
	for (int i = 0; i < GETC_REPS; tester_buff++, i++) {
	TEST_ASSERT_EQUAL(tester_buff, td.rx_buff[i]);
	}
	// Make sure TX IRQ was disabled during the last RX test.
	TEST_ASSERT_EQUAL_UINT32(checksum, tester.rx_get_checksum());
	TEST_ASSERT_EQUAL_UINT32(2 * PUTC_REPS, tester.rx_get_count());
	// Cleanup

Mutex: 0x8003B14, Not allowed in ISR context
This test PASSED if comment line 251:

mbed-os/TESTS/mbed_hal_fpga_ci_test_shield/uart/main.cpp

Line 251 in 6993724

serial_irq_set(&serial, RxIrq, 1);

It looks like something bad happens inside serial interrupt handler - some disallowed operation and we got an exception. serial_irq_set() enables serial interrupts, so if you remove this line probably interrupt mode is disabled, so you don't have the exception.

@fkjagodzinski have you seen such behavior?

[yarbcy]

@mprse

So if you run only one test case (any) it fails randomly?

Yes.

Comment line 252 - 271
What does it mean? What is the purpose of removing these lines:

It is not important. I commented because I wanted to isolate the problem.

[yarbcy]

@fkjagodzinski ?

[fkjagodzinski]

3. Comment line 252 - 271

++ MbedOS Error Info ++
[1573648317.62][CONN][RXD] Error Status: 0x80010133 Code: 307 Module: 1
[1573648317.68][CONN][RXD] Error Message: Mutex: 0x8003B14, Not allowed in ISR context
[1573648317.70][CONN][RXD] Location: 0x100095AD
[1573648317.73][CONN][RXD] Error Value: 0x8003B14
[1573648317.84][CONN][RXD] Current Thread: main Id: 0x8003B38 Entry: 0x10009477 StackSize: 0x1000 StackMem: 0x8004A18 SP: 0x80055D4
[1573648317.94][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80010133&tgt=CY8CKIT_062_WIFI_BT

This looks suspicious. On current master (and also the SHA you mentioned -- f1848f9) you should not remove line 252 alone; you should also remove 249 & 251.

mbed-os/TESTS/mbed_hal_fpga_ci_test_shield/uart/main.cpp

Lines 249 to 252 in 2e96145

	core_util_critical_section_enter();
	// Enable only the RX IRQ.
	serial_irq_set(&serial, RxIrq, 1);
	core_util_critical_section_exit();

I do not have any Cypress target with an Arduino form factor to test with FPGA shield but I can see a similar crash on K64F if I comment out 252-271:

[1574070560.88][CONN][RXD] ++ MbedOS Error Info ++
[1574070560.92][CONN][RXD] Error Status: 0x80010133 Code: 307 Module: 1
[1574070560.98][CONN][RXD] Error Message: Mutex: 0x2000113C, Not allowed in ISR context
[1574070560.99][CONN][RXD] Location: 0x6817
[1574070561.03][CONN][RXD] Error Value: 0x2000113C
[1574070561.13][CONN][RXD] Current Thread: main Id: 0x20001160 Entry: 0x6747 StackSize: 0x1000 StackMem: 0x200022B0 SP: 0x20002F50
[1574070561.21][CONN][RXD] For more info, visit: https://mbed.com/s/error?error=0x80010133&tgt=K64F
[1574070561.23][CONN][RXD] -- MbedOS Error Info --

If you like to skip DUT RX (IRQ) / FPGA TX part of the test, I suggest commenting out all the related code (lines 239-269):

mbed-os/TESTS/mbed_hal_fpga_ci_test_shield/uart/main.cpp

Lines 239 to 269 in 2e96145

	// DUT RX (IRQ) / FPGA TX
	// serial_getc() may return 16-bit as well as 8-bit value cast to an int.
	// Use a random initial value, but make sure it is low enouth,
	// so the FPGA will not overflow 8 bits when incrementing it.
	tester_buff = rand() % ((1 << test_buff_bits) - GETC_REPS);
	tester.tx_set_next(tester_buff);
	tester.tx_set_count(GETC_REPS);
	if (!use_flow_control) {
	tester.tx_set_delay(TX_START_DELAY_NS);
	}
	core_util_critical_section_enter();
	// Enable only the RX IRQ.
	serial_irq_set(&serial, RxIrq, 1);
	core_util_critical_section_exit();
	tester.rx_start();
	tester.tx_start(use_flow_control);
	while (core_util_atomic_load_u32(&td.rx_cnt) != GETC_REPS) {
	// Wait until the last byte is received to UART RX reg.
	};
	core_util_critical_section_enter();
	serial_irq_set(&serial, RxIrq, 0);
	core_util_critical_section_exit();
	tester.tx_stop();
	tester.rx_stop();
	for (int i = 0; i < GETC_REPS; tester_buff++, i++) {
	TEST_ASSERT_EQUAL(tester_buff, td.rx_buff[i]);
	}
	// Make sure TX IRQ was disabled during the last RX test.
	TEST_ASSERT_EQUAL_UINT32(checksum, tester.rx_get_checksum());
	TEST_ASSERT_EQUAL_UINT32(2 * PUTC_REPS, tester.rx_get_count());

[yarbcy]

@mprse @0xc0170 Please take a look @fkjagodzinski can reproduce this issue:

I do not have any Cypress target with an Arduino form factor to test with FPGA shield but I can see a similar crash on K64F if I comment out 252-271:

[mprse]

I think that we are missing the point here. @fkjagodzinski reproduced your failure on K64F by removing some random part of the test that you mentioned. Original test version passes on the K64F.

I think that we should start again since I think that no one understands now where or even if we have a problem.

Please provide full test output with the failure (without modifying the test).