Hard fault: memory access or instruction error found intermittently on RP2040 CAN Bus Feather

[0nix]

CircuitPython version

8.2.7

Code/REPL

RCVRID = 0x001 #CHANGE THIS LINE
# RP2040 CAN Bus Feather

from time import sleep
import time
import board
import neopixel
from rainbowio import colorwheel
from digitalio import DigitalInOut, Pull
from adafruit_mcp2515.canio import Message, RemoteTransmissionRequest
from adafruit_mcp2515 import MCP2515 as CAN
import adafruit_mcp2515.canio as canio

COLORS = {
    0x0: (0,0,0,0),
    0x1: (255,0,0,0),
	0x2: (0,255,0,0),
	0x3: (0,0,255,0),
	0x4: (255,255,0,0),
	0x5: (0,255,255,0),     
	0x6: (255,0,255,0),
	0x7: (0,0,0,255),
	0x8: (255,128,0,0),
	0x9: (0,128,255,0),
	0xA: (128,0,255,0), 
	0xB: (255,0,128,0), 
	0xC: (0,128,0,0),
	0xD: (100,149,237,0),
	0xE: (139,0,0,0),
	0xF: (218,165,32,0),
	0x10: (32,178,270,0),
	0x11: (255,69,0,0),
	0x12: (0,255,255,128),
    0x13: 0x13,  #Rainbow 		= 0x13, 
    0x14: 0x14 #FastRainbow	= 0x14,
    #Skip			= 0xA0
} 
GREEN = (0,255,0)
RED = (255,0,0)
WHITE = (255,255,255)
NONE = (0,0,0)

LLPin = DigitalInOut(board.D5)
ULPin = DigitalInOut(board.D6)
URPin = DigitalInOut(board.D9)
LRPin = DigitalInOut(board.D10)

LL_LightOn = COLORS[0x3]
LL_LightOff = COLORS[0x0]
UL_LightOn = COLORS[0x3]
UL_LightOff = COLORS[0x0]
UR_LightOn = COLORS[0x3]
UR_LightOff = COLORS[0x0]
LR_LightOn = COLORS[0x3]
LR_LightOff = COLORS[0x0]

RainbowColor = 0x0
FastRainbowSet =  ((255, 0, 0, 0), (255, 40, 0, 0), (255, 150, 0,0), (0, 255, 0,0), (0, 0, 255, 0), (180, 0, 255,0))

LightsPin = board.D11

Lights = neopixel.NeoPixel(LightsPin, 4, brightness = 0.5, auto_write = True, pixel_order = neopixel.RGBW)
statusPixel = neopixel.NeoPixel(board.NEOPIXEL, 1)
statusPixel.brightness = 0.1

try:
    cs = DigitalInOut(board.CAN_CS)
    cs.switch_to_output()
    spi = board.SPI()

    can_bus = CAN(
        spi, cs, loopback=False, silent=False
    )  # use loopback to test without another device
except Exception as e:
    print(e)
    statusPixel.fill((255,0,255))

LLPin.switch_to_input(pull=Pull.UP)
ULPin.switch_to_input(pull=Pull.UP)
URPin.switch_to_input(pull=Pull.UP)
LRPin.switch_to_input(pull=Pull.UP)

DEBUG = False
HAS_INITED = False
MSGTIMEOUT = 1

def can_send_msg(sender_id,data):
    message = canio.Message(id=sender_id, data=data)
    can_bus.send(message)
    if (can_bus.transmit_error_count > 0) or (can_bus.receive_error_count > 0):
        print(
            f"🔴 MSG tx_err={can_bus.transmit_error_count} rx_err={can_bus.receive_error_count}"
        )
    return

def can_recv_msg():
        msg = listener.receive()
        if (can_bus.transmit_error_count > 0) or (can_bus.receive_error_count > 0):
            print(f"🔴 MSG tx_err={can_bus.transmit_error_count} rx_err={can_bus.receive_error_count}")
        if msg is None:
            if DEBUG: print("🟡 MSG not received within timeout")
            return
            #continue
        if isinstance(msg, canio.Message):
            print(f"RCCV = {msg.data}")
            if(msg.id == RCVRID + 0X100):
                try:
                    global LL_LightOn
                    global LL_LightOff
                    global UL_LightOn
                    global UL_LightOff
                    global UR_LightOn
                    global UR_LightOff
                    global LR_LightOn
                    global LR_LightOff

                    lights = list(msg.data)
                    print(f"rcv = LIGHTS {lights}")
                    for idx, x in enumerate(lights):
                        if lights[idx] == 0xA0:
                            continue
                        if idx == 0:
                            LL_LightOn      = COLORS[lights[0]]
                        if idx == 1:
                            LL_LightOff     = COLORS[lights[1]]
                        if idx == 2:
                            UL_LightOn      = COLORS[lights[2]]
                        if idx == 3:
                            UL_LightOff     = COLORS[lights[3]]
                        if idx == 4:
                            UR_LightOn      = COLORS[lights[4]]
                        if idx == 5:
                            UR_LightOff     = COLORS[lights[5]]
                        if idx == 6:
                            LR_LightOn      = COLORS[lights[6]]
                        if idx == 7:
                            LR_LightOff     = COLORS[lights[7]]
                except Exception as e:
                    print(e)
        else:
            if DEBUG: print("🟡 not a canio message")


def setLight(lightArr,idx,colorRef):

    if(colorRef == 0x13): #Rainbow
        lightArr[idx] = RainbowColor
        return
    if(colorRef == 0x14): #Fast Rainbow
        lightArr[idx] = FastRainbowColor
        return
    if(lightArr[idx] != colorRef):
        lightArr[idx] = colorRef

matcher = canio.Match(address=(RCVRID + 0X100))
listener = can_bus.listen(timeout=MSGTIMEOUT, matches=[matcher])

old_bus_state = None
statusPixel.fill((0,0,0))
global timestamp
timestamp = time.monotonic()
lastTick = -1
lastFrame = 0
lastFastTick = -1
lastFastFrame = 0
readInterval = 1
fastReadInterval = 0.083
lastMsg = ""
while True:
    statusPixel.fill((0,0,0))
    now = time.monotonic()
    bus_state = can_bus.state
    if bus_state != old_bus_state:
                if bus_state == 0:
                    print(f"🟣 BUS state changed to ACTIVE")
                if bus_state == 1:
                    print(f"🟣 BUS state changed to WARNING -- MODERATE ERRORS")
                if bus_state == 2:
                    print(f"🟣 BUS state changed to PASSIVE -- TOO MANY ERRORS")
                if bus_state == 3:
                    print(f"🟣 BUS state changed to OFF -- NO ACK ON PACKETS")
                old_bus_state = bus_state
    if bus_state == 0 and HAS_INITED == False:
        Lights.fill((0,0,0,0))
        Lights[0] = (255,0,0)
        Lights[1] = (0,255,0)
        Lights[2] = (0,0,255)
        Lights[3] = (255,255,255)
        HAS_INITED = True
        sleep(2)
        Lights[0] = (0,0,0)
        Lights[1] = (0,0,0)
        Lights[2] = (0,0,0)
        Lights[3] = (0,0,0)

    if bus_state == 1:
        statusPixel.fill((255,255,0))
    if bus_state == 2:
        statusPixel.fill((255,0,0))
    if bus_state == 3:
        statusPixel.fill((255,255,255))

    RainbowColor = colorwheel((time.monotonic()*50)%255)
    
    if now >= lastFastTick + fastReadInterval:
        lastFastTick = now
        if(lastFastFrame < len(FastRainbowSet) - 1):
            lastFastFrame += 1
        else:
            lastFastFrame = 0

        FastRainbowColor = FastRainbowSet[lastFastFrame]

    msg = "="
    
    if(LLPin.value == False): # D5 Pin
        msg += "a"
        setLight(Lights,0,LL_LightOn)
    else:
        msg += "x"
        setLight(Lights,0,LL_LightOff)
    if(ULPin.value == False): # D6 Pin
        msg += "b"
        setLight(Lights,1,UL_LightOn)
    else:
        msg += "x"
        setLight(Lights,1,UL_LightOff)
    if(URPin.value == False): # D9 Pin
        msg += "c"
        setLight(Lights,2,UR_LightOn)
    else:
        msg += "x"
        setLight(Lights,2,UR_LightOff)
    if(LRPin.value == False): #D10 Pin
        msg += "d"
        setLight(Lights,3,LR_LightOn)
    else:
        msg += "x"
        setLight(Lights,3,LR_LightOff)
    
    try:
        if DEBUG: print(f"MSG avail={listener.in_waiting()} unread={can_bus.unread_message_count}")
        if bus_state == 0 or bus_state == 1:
            if(can_bus.unread_message_count > 0):

                print('process can')
                lastRead = now
                can_recv_msg()

            if(lastMsg != msg):
                can_send_msg(RCVRID,bytes(msg,'ascii'))
                
                lastMsg = msg
        else:
            if (msg == "=xxxx"):
                pass
            else:
               print(msg)
               statusPixel.fill((0,0,255))
            
    except Exception as e:
        print(e)
        statusPixel.fill((255,255,0))

Behavior

Code runs normally until arbitrarily, it crashes, stops updating the NeoPIxel strip based on pin status, and presents the error You are in safe mode because: CircuitPython core code crashed hard. Whoops! Hard fault: memory access or instruction error.

Description

No response

Additional information

This bug report was submitted under advisement of Dan Halbert on the Discord server.

I understand the error to be a memory access or allocation error.

I don't know why this happens and how to get it to appear consisntely. I've got it running in 50+ different instances of the same board with the same CircuitPython version and it never happens on some boards, and on others it just happens by leaving the board to be idle.

I suspect that my allocation of strings that constitute the message to be written to the CAN Bus to be culprit and have ran a modified version of the code that uses a list of characters instead of writing a string.

# previous code here
msgSeg = ['=','x','x','x','x']

while True:
# some code here
    if(LLPin.value == False): # D5 Pin
        msgSeg[1]= "a"
        setLight(Lights,0,LL_LightOn)
    else:
        msgSeg[1]= "x"
        setLight(Lights,0,LL_LightOff)
    if(ULPin.value == False): # D6 Pin
        msgSeg[2]= "b"
        setLight(Lights,1,UL_LightOn)
    else:
        msgSeg[2]= "x"
        setLight(Lights,1,UL_LightOff)
    if(URPin.value == False): # D9 Pin
        msgSeg[3]= "c"
        setLight(Lights,2,UR_LightOn)
    else:
        msgSeg[3]= "x"
        setLight(Lights,2,UR_LightOff)
    if(LRPin.value == False): #D10 Pin
        msgSeg[4]= "d"
        setLight(Lights,3,LR_LightOn)
    else:
        msgSeg[4]= "x"
        setLight(Lights,3,LR_LightOff)
    msg = ''.join(msgSeg)
    try:
        if DEBUG: print(f"MSG avail={listener.in_waiting()} unread={can_bus.unread_message_count}")
        if bus_state == 0 or bus_state == 1:
            if(can_bus.unread_message_count > 0):

                lastRead = now
                can_recv_msg()

            if(lastMsg != msg):
                can_send_msg(RCVRID,bytes(msg,'ascii'))
                
                lastMsg = msg
        else:
            if (msg == blankMsg):
                pass
            else:
               #print(msg)
               statusPixel.fill((0,0,255))
            
    except Exception as e:
        print(e)
        statusPixel.fill((255,255,0))

# rest of code

This has not been as consistently tested as the previous string approach, but it is showing some promise. This does not explain how some boards are able to deal with this issue just fine while others do not.

Furthermore, the boards may be exposed to a higher chance of ESD since they are in proximity to acrylic boards where people are expected to step on and slide. While the acrylic does not come into direct contact with the boards, I have noticed they accumulate static electricity so this may also play a factor.

The code also does not sleep between while loops, would this be a factor?

Any and all help is tremendously appreciated.

[bill88t]

You may want to test 9.0-alpha builds, since the memory management system changed a good bit.

[0nix]

Would you advise the use of 9.0-alpha on a production setting with code like this? It is not going to change in functionality.

[dhalbert]

Testing the alpha build might give us some clues about what the problem is. I don't think we would necessarily advise it for use in production. but if it works better than 8.2.x that is a selling point. "Stable" is in the eye of the beholder. Alpha builds are generally not deliberately unstable.

[0nix]

This script lacks a sleep at the end of the loop. Would that improve anything? All examples include a call to sleep, and I assume this is a best practice scenario.

[anecdata]

Senders may typically have a delay between messages, but if the device is primarily listening for messages, there should be no delay and you want to minimize the other processing in the main loop. There's not much buffering in the CAN hardware (2 receive buffers and 3 send buffers), so loops need to complete in a timely manner relative to the frequency of incoming messages.

fwiw, I have yet to see a safemode on CAN projects (though with small number of devices on the bus), it's possible the fault is due to something else

[tannewt]

Please retest this with an absolute newest build and try to trim down the test code. I was unable to reproduce this with a disconnected bus and no other devices.

[SamantazFox]

Could it be related to adafruit/Adafruit_CircuitPython_MCP2515#28 ?