Dropped CAN frames using Socketcan on Raspberry Pi

[Lauszus]

I'm using this library for logging data sent on a CAN-Bus running at 500 kbit/s. The data is sent roughly at 1 kHz.

I have set up a CAN Notifier to receive the data in a thread and the data is then written to a queue in a listener:

class LoggerBufferedListener(can.Listener):
    def __init__(self, log_opened_event: Event):
        self._log_opened_event = log_opened_event
        self._buffer = Queue()
        self._lock = Lock()
        self._closed = False

    def set_closed(self, closed):
        with self._lock:
            self._closed = closed

    def is_closed(self):
        with self._lock:
            return self._closed

    def on_message_received(self, msg: can.Message):
        # Only put the CAN-Bus message into the buffer if it is a close/open command or we are currently logging data
        # and the arbitration ID matches any of the logger IDs
        if msg.arbitration_id == CANLogger.ID_CLOSE_OPEN or \
                (self._log_opened_event.is_set() and
                 msg.arbitration_id in [logger['id'] for logger in CANLogger.Loggers]):
            self.put_nowait(msg)

    def get(self):
        return self._buffer.get()

    def get_nowait(self):
        try:
            return self._buffer.get_nowait()
        except queue.Empty:
            return None

    def put(self, item, force=False):
        if not force:
            if self.is_closed():
                return  # Return if the reader is closed

        self._buffer.put(item)

    def put_nowait(self, item):
        if self.is_closed():
            return  # Return if the reader is closed

        try:
            self._buffer.put_nowait(item)
        except queue.Full:
            Logger.exception('LoggerBufferedListener: Buffer is full')

Another thread is doing nothing else than calling the get() method, the messages are then unpacked and converted to SI-units similar to how I did it in the viewer program:

python-can/can/viewer.py

Lines 135 to 159 in 9f6f918

	# Unpack the data and then convert it into SI-units
	@staticmethod
	def unpack_data(cmd, cmd_to_struct, data): # type: (int, Dict, bytes) -> List[Union[float, int]]
	if not cmd_to_struct or len(data) == 0:
	# These messages do not contain a data package
	return []
	for key in cmd_to_struct.keys():
	if cmd == key if isinstance(key, int) else cmd in key:
	value = cmd_to_struct[key]
	if isinstance(value, tuple):
	# The struct is given as the fist argument
	struct_t = value[0] # type: struct.Struct
	# The conversion from raw values to SI-units are given in the rest of the tuple
	values = [d // val if isinstance(val, int) else float(d) / val
	for d, val in zip(struct_t.unpack(data), value[1:])]
	else:
	# No conversion from SI-units is needed
	struct_t = value # type: struct.Struct
	values = list(struct_t.unpack(data))
	return values
	else:
	raise ValueError('Unknown command: 0x{:02X}'.format(cmd))

The data is then written to RAM. After the logged is closed the data is then stored in an Influx database.

However it seems that messages are dropped somewhere, as the data often has gaps in it.

I'm using a MCP2515 and have enabled it using the following overlays in /boot/config.txt:

dtoverlay=mcp2515-can0,oscillator=16000000,interrupt=12
dtoverlay=spi0-hw-cs

My question is basically if anyone has any experience using Socketcan on a Raspberry Pi. Do you guys believe it's a hardware or software issue? I've considered using a different CAN tranceiver than the MCP2515 just to check if that is the culprit. If that does not work I might try to implement the logging in C instead, but would prefer to use Python, as that makes all the parsing of data and writing to the database much easier.

[teebr]

I gave it a try with a 1Mbps bus averaging around 4 frames / ms and had a lot of dropped messages. I didn't look into it too much (tried a few variants of threading/ multiprocessing, and reduced the code to just reading as frequently as possible - no data processing), I figured Python was just not able to clear the buffer quick enough.

Instead, I wrote a small C client which dumps the raw CAN data to a server, and did the processing on there (coincidentally I was also using Influx, so just used the same server that was running on)

[christiansandberg]

Have you tried candump?

[Lauszus]

@teebr thanks for the reply. I might give that a try. Is your code open source?

@christiansandberg no I have not.

[teebr]

@Lauszus It's not at the moment- it was a work thing and honestly it was very hacky, I'll see if I can make it into a gist though. In the meanwhile you can use the candump.c source as a basis for how to read from the socket(s).

I had two threads running: one polling for CAN messages and putting the frame and timestamp in a char array, and the other just used memcpy (with a mutex) to copy the buffer over and send it to a local server. The most lightweight way would just be to open a TCP socket with the server and continually dump bytes. Using libcurl to send the bytes over POST is maybe a bit more robust, but there's more overhead.

[Lauszus]

@teebr okay thanks! I'll give that a try.

[TravisJoe]

I have had good luck with spawning a CAN manager processes, to have a truly independent threads. Then using Queues to push outgoing messages and to receiving queue for incoming messages after they have been formatted and timestamped.

[Lauszus]

@TravisJoe I'm not sure what you mean? The Notifier class is launched in a thread and as you can see in my code above my listener does nothing else than just write the incoming messages in a queue.

[TravisJoe]

@Lauszus
Without knowing all the code I am assuming you are using threads and not processes. Since in a python script is really just a single processes you still end up sharing resources when threading. Spawning a processes for only does your listener class would likely help.

Also hardware filtering is the best only way to reduce overhead, second best is filtering at the lower OS level sockets. Filtering in code is about as slow as you can get, especially with python creating new objects every time. Use the hardware or at least the lower level socket filters and avoid the "if" statement in a listener. If you have to filter do it outside of the listener after it is placed in a queue.

You can also use a dictionary like a C style switch statement which is better than multiple if statements when speed in python is needed.

You might also be able to increase the socket buffer size.

[FabianInostroza]

Hi @Lauszus, did you solve the problem?

I'm using an MCP25625 with an ARM board (slower than the RPi, only 400MHz) in a bus with about 1200 frames/s.
I also notice there are some frames dropped and I found that python uses a lot of cpu (>60%) when the script being run connects to the can bus, even if the script does nothing with the frames.

I wrote a C program that uses BCM to downsample the frames and send them to a vcan.
With that scheme the python script runs better in term of cpu usage (I haven't verified the frame drop, I was worried about cpu usage). This C program uses about 1% cpu.

[Lauszus]

@TravisJoe I've tried using processes as well with the same result. I do use hardware filtering using the can_filters argument. From what I have read the difference of using a dict instead of if statements is really neglectable.

Thanks, I'll try increased the socket buffer size!

@FabianInostroza thanks for the input. My problem is that I actually do not want to downsample it, as I am logging some data via CAN-Bus. It's starting to sound like the best way to solve this was just to write the logging code in C.

[teebr]

@Lauszus I fleshed out my original comment into a small package. You run Python stuff on a local server and a small C client on the Linux device to send/ receive on the CAN bus. There's a raw TCP connection between the two, so latency is low. CPU usage on a Raspberry Pi 3B is < 10% with 2000 messages / sec. Have a look, it might suit your needs or give you some ideas.

[Lauszus]

@teebr I just tested it and it works very well!

I have added pybind11 bindings to the client: teebr/socketsocketcan#1. Let's move the discussion to your repo :)

[thbuch]

Hi,
I am noticing the same frames missing when trying to log to blf.
I tested with candump logging to log format and the same result.
I tried to expand input buffer using:
$echo 100048576 | sudo tee /proc/sys/net/core/rmem_max
$echo 100048576 | sudo tee /proc/sys/net/core/rmem_default

Initial buffer was 1048576 but I still got missing messages.

Have you experienced that problem using a Raspberry Pi and the Seeed Studio 2CH CAN FD Hat?
At lower data transfer speed it looks fine.

[Lauszus]

No I have not used that particular hat. FYI my repo is here in case you want to use it: https://github.com/Lauszus/socketsocketcan.

I've been using it for years now without any issues with dropped frames.