test: tests/kernel/mem_slab/mslab_threadsafe fails sporadically on nrf52

[mike-scott]

Describe the bug
If you run the following test on nrf52_blenano2 (and possibly any nrf52832) it will fail with a hardware exception in a seemingly random pattern.

Painful repeated bisection leads me to this commit:
03f007e

To Reproduce
Steps to reproduce the behavior:

1. cd tests/kernel/mem_slab/mslab_threadsafe
2. mkdir build; cd build
3. cmake -DBOARD=nrf52_blenano2 ..
4. make flash
5. 1 in say 6-7 tries (sometimes a lot more) see hardware exception

Expected behavior
Pass this test

Impact
CI fails in random ways when testing on real HW

Screenshots or console output

***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
PASS - test_mslab_threadsafe                                                                                                                                 
===================================================================                                                                                          
Test suite mslab_threadsafe succeeded                                                                                                                        
===================================================================                                                                                          
PROJECT EXECUTION SUCCESSFUL                                                                                                                                 
***** Booting Zephyr OS v1.14.0-rc1-330-gb71e1f941f *****                                                                                                    
Running test suite mslab_threadsafe                                                                                                                          
===================================================================                                                                                          
starting test - test_mslab_threadsafe                                                                                                                        
***** MPU FAULT *****                                                                                                                                        
  Data Access Violation                                                                                                                                      
  MMFAR Address: 0x0                                                                                                                                         
***** Hardware exception *****                                                                                                                               
Current thread ID = 0x20000114                                                                                                                               
Faulting instruction address = 0x4436                                                                                                                        
Fatal fault in ISR! Spinning...                                                                                                                              

Environment (please complete the following information):

• OS: Arch Linux
• Toolchain: Zephyr SDK
• Master as of today (SHA: b71e1f9)

[andyross]

Immediate things to check would be to (1) disable CONFIG_TICKLESS_KERNEL to see if we can narrow down the issue to the driver core or tickless mode, both of which changed with that commit, and (2) crank up the size of the ISR stack as a quick check to rule out simple stack overflow ("random corruption" always sounds like stack overflow to me, heh)

[mike-scott]

1. CONFIG_TICKLESS_KERNEL=n doesn't affect. I was able to reproduce immediately.

[mike-scott]

2. I bumped CONFIG_ISR_STACK_SIZE to 4096 (up from 2048) and was also able to reproduce.

[mike-scott]

For more detail on the HW exception faulting instruction:
/zephyr/include/misc/dlist.h:523

[andrewboie]

Same issue perhaps as #12559 ?

[andyross]

Yeah, this was definitely due to the issue in #13770. Just ran through 80 iterations of this test on my nrf52840_pca10056 and my finger is sore from the surprisingly sharp reset button.

[andyross]

Fixed in the tree for me. Please reopen if needed.