sysex message size limit when receiving

[Remifa2]

Hi

I've a problem with receiving sysex messages larger than 1024 bytes. Sending large sysex messages up to 200kB works properly. I'm not aware of any restrictions apart from RAM that would limit the length of sysex messages.

Does anyone have the same problem and how do I solve this problem?

I'm working on Windows 10 with Python 3.

Thanks in advance for any help.

[Remifa2]

Can anyone test this, please?

[rdoursenaud]

Hi,

Single SysEx message receive size can be limited by quite a number of factors:

• The MIDI interface hardware buffer
• Its driver implementation
• The operating system
• mido’s backend

Do you have the same behavior using a DAW?

[Remifa2]

Thanks for your attention. I'll give some details of the context I'm working in.

I'm using loopMIDI and Windows 10. I'm not using any other hardware than my Windows PC.
I've been using the mido interface in SWI Prolog. More precisely I'm using the janus-swi interface between SWI Prolog and Python.

Here is the thread in the SWI Prolog forum where I reported the issue initially : https://swi-prolog.discourse.group/t/janus-message-size-limit/7704, suspecting the swi-janus interface was the cause of the issue.
As long as the SYSEX communication doesn't work I'm stuck.

[rdoursenaud]

Your culprit is loopMidi then. It has a setting in the Advanced tab for Maximum Sysex-size.

[Remifa2]

The setting in loopMIDI is 256 kilobyte, i.e. 256 times 1024 bytes. I don't know when it was set or whether it's the default setting. Anyway, why would loopMIDI discriminate between sysex messages coming from or going to mido?

[rdoursenaud]

Hum, I reproduced your issue and traced it back to rtmidi.
While its default buffer size is 1024 bytes, python-rtmidi - which is the wrapper we use as the default mido backend - should support up to 8196.
However it seems there’s a bug in recent versions.
Could you try reverting to python-rtmidi 1.4.9 as stated in the last comment?

[Remifa2]

Thanks for checking and finding that post.
I wonder why there exist any limits besides RAM memory limits for MIDI sysex messages. My RAM is 8 GB large. 8 kB is a serious limit as well and I'm not going to change anything to my 'stable' configuration. Rtmidi can send very large messages, but not receive them. IMO, this is absurd. I'll wait for a real solution to this problem.

[rdoursenaud]

It all seems to come from a misinterpretation of the specification IMHO.
For me SysEx messages should have been treated as "Running status" messages and delivered byte-by-byte.

However, quite a lot of implementations have put arbitrary limits on SysEx receive size because they buffer bytes before delivering them as a single "packetized" message instead of streaming the data.
As stated in my first reply, I have seen it in hardware, drivers, operating systems, various libraries and applications even.

I concur it is an anachronistic and arbitrary limitation.
I’m trying to get away from some abstractions for mido 2.0 and provide more flexibility in this regard:

• "Native" backends that uses the operating system’s abstractions directly
• and maybe low level backends (USB, UART, SERIAL-MIDI)

Unfortunately I can’t work on it currently because I’ve been moving my life elsewhere and things got in the way with everything stuck in limbo for now. I hope to recover my capacity to work in a few months.