Running status issue? Midi messages mangled and subsequent communication broken/out of sync.

[hallvardkristiansen]

Context

Using the English language, please answer a few questions to help us understand your problem better and guide you to a solution:

• What board are you using ?
  • 7 x Teensy 3.6
  • Wiznet W5500 ethernet chip
  • MCP23017 multiplexers
• What version of the Arduino IDE are you using ?
  • Teensyduino 1.8.15
• What Operating System are you using for the DAW
  • Windows
  • MacOS
  • Linux
  • Other (please specify)
• Is your problem related to:
  • Setup
  • Connecting
  • AppleMIDI <> MIDI
  • Dropped messages
• How comfortable are you with code ?
  • Complete beginner
  • I've done basic projects
  • I know my way around C/C++
  • Advanced / professional

Describe your project and what you expect to happen:

The Teensies are controlling lights and buttons on a sequencer, controlled by a computer running PD, Ableton and MidiPipe. The Teensies connect automatically using sendInvite in a staggered sequence which works fine. The Teensies send and receive midi without issue for the most part. Midi notes are mapped to particular LEDs and switches.

Describe your problem (what does not work):

Perhaps "bug" is not correct here, but I don't have any better way of describing the issue. Midi messages sent by MidiPipe to reset LEDs and other functions get dropped by the Teensy. When sending eight messages with noteOff, only two arrive correctly and occasionally a third arrives as noteOn. Sometimes when spamming a button, the messages can make it through, but this only happens in one out of 20 attempts.

Steps to reproduce

Send batches of the same type of message to the Teensy using MidiPipe.
We have checked all the hardware, tested replacing the Teensy with a macbook which received all the messages fine on the same ethernet cable. We also checked serial output of raw midi data which looked correct.
I have a suspicion that we are dealing with running status messages not being parsed by the Teensy, and thus making the subsequent messages nonsense.

Right now I do not have direct access to the hardware, but I could request a serial dump of the raw midi messages as soon as possible.

Would appreciate any suggestions for solutions!

[lathoub]

Thanks @hallvardkristiansen , sounds like a buffer overrun/flow that occurred before: #100 Quite a long thread, but from what you describe, you could be hitting the same issue (DAW's typically generate a lot of background reset messages, combined with 'spamming a button' (hitting a button repeatedly, like in the Hyper Olympics arcade :-))

Here is a summary of the buffer overflow issue (and how to solve): https://github.com/lathoub/Arduino-AppleMIDI-Library/wiki/Enlarge-Ethernet-buffer-size-to-avoid-dropping-UDP-packages

If this is not the issue, then a serial dump is needed.

[hallvardkristiansen]

Hi! Thanks for the fast reply!
Yes that is what I thought too. But I read the documentation so I was aware of this and tried the ethernet3 library instead.
This didn’t change anything regarding incoming messages(exact same behavior), but it caused the teensy to lock up when it sends a midi message. I will ask for a serial dump of the issue and post it here asap.

I suppose in order to enable running status support I would need to initialize the midi class separately first and then do a custom initialization of the applemidi class? It should pass the modified midi reference to applemidi then, correct?

[lathoub]

Running status is enabled (always) as part of parsing incoming AppleMIDI messages, but I noticed that Running Status is not on in the Settings of the underlying MIDI library, see https://github.com/FortySevenEffects/arduino_midi_library/blob/d501f4bb3dfff548a7cef4d85371bec0b438bdd7/src/midi_Settings.h#L51-L56

So you could override the Setting when creating the MIDI object, see

Arduino-AppleMIDI-Library/src/AppleMIDI.h

Lines 339 to 341 in 1f095b7

	#define APPLEMIDI_CREATE_INSTANCE(Type, Name, SessionName, Port) \
	APPLEMIDI_NAMESPACE::AppleMIDISession<Type> Apple##Name(SessionName, Port); \
	MIDI_NAMESPACE::MidiInterface<APPLEMIDI_NAMESPACE::AppleMIDISession<Type>, APPLEMIDI_NAMESPACE::AppleMIDISettings> Name((APPLEMIDI_NAMESPACE::AppleMIDISession<Type>&)Apple##Name);

[hallvardkristiansen]

Thank you. I will try writing a custom initialization that enables running status in the midi library. Meanwhile my colleague is extracting a serial dump of the raw midi when the issue is occurring. Will post it here as soon as it arrives.

[lathoub]

if you can, please provide the dump as:

byte buffer[] = {
                 0x80, 0x61, 0xbf, 0xa2, 0x12, 0xb, 0x5a, 0xf7, 0xaa, 0x34, 0x96, 0x4a,
                 0xc0, 0x2b,
                 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x0, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x0,
                 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x00, 0xf8, 0x0, 0xf8, 0x00, 0xf8, 0xc0, 0xbf, 0x89, 0x90, 0x05, 0xd0, 0x7a, 0xd5 };

(I'll use using the xcodeproj project in the /test dir to do the debugging)

[hallvardkristiansen]

Here is one of the latest outputs that has been cut of in the middle.

byte buffer[] = {0x92, 0x70, 0x00, 0x71, 0x00, 0x72, 0x00, 0x73, 0x00, 0x74, 0x00, 0x9A, 0x60, 0x78, 0x61, 0x00, 0x61, 0x78, 0x60, 0x00, 0xBA, 0x33, 0x32, 0x94, 0x71, 0x7F, 0x72, 0x00, 0x73, 0x00, 0x75, 0x7F, 0x76, 0x00, 0x94, 0x70, 0x00, 0x74, 0x00, 0x78, 0x00, 0x90, 0x70, 0x00, 0x92, 0x70, 0x00, 0x71, 0x00, 0x72, 0x00, 0x73, 0x00, 0x74, 0x00, 0xBA, 0x33, 0x32, 0x94, 0x71, 0x7F, 0x72, 0x00, 0x73, 0x00, 0x75, 0x7F, 0x76, 0x00, 0x77, 0x00, 0x79, 0x7F, 0x7A, 0x00, 0x7B, 0x00, 0x70, 0x00, 0x74, 0x00, 0x78, 0x00, 0x90, 0x71, 0x00, 0x92, 0x70, 0x00, 0x71, 0x80, 0x92, 0x74, 0x00, 0x90, 0x70, 0x78, 0x71, 0x78, 0x72, 0x00, 0x73, 0x00, 0x74, 0x00, 0x75, 0x00, 0x76, 0x00, 0x77, 0x00, 0x70, 0x00, 0x71, 0x00, 0x72, 0x00, 0x73, 0x00, 0x74, 0x00, 0x75, 0x00, 0x76, 0x00, 0x94, 0x71, 0x00, 0x72, 0x7F, 0x73, 0x00, 0x94, 0x70, 0x78, 0x75, 0x00, 0x76, 0x7F, 0x77, 0x00, 0x79, 0x00, 0x7A, 0x7F, 0x7B, 0x00, 0x74, 0x78, 0x78, 0x78, 0x70, 0x00, 0x74, 0x00, 0x78, 0x00, 0xBA, 0x32, 0x28, 0x90, 0x72, 0x00, 0x90, 0x73, 0x00, 0x90, 0x74, 0x00, 0x90, 0x75, 0x00, 0x90, 0x76, 0x00, 0x90, 0x77};

Will that work for you?
It seems it contained about 500 bytes of actual data values, the other characters would maybe push that over the default buffer size? The messages are definitely running status style, but they do at least for the most part seem to be handled well by the teensy.

[lathoub]

Thanks - this looks like a raw MIDI stream, not AppleMIDI.
Can you provide the Wireshark log?

[hallvardkristiansen]

Right, we misunderstood.
I’ll try to get the wireshark log.
Thanks

[hallvardkristiansen]

Problem-Occurs-afte-7-sec.pcapng.zip

Here is the log I received. It shows that the issues begin when the RTP-MIDI messages start arriving from the computer(10.0.0.6) from message nr. 1015

The RTP type messages are coming from PureData, while the RTP-MIDI messages come from Ableton via MidiPipe.
It's not obvious to me why this would cause a problem, but maybe you see something here?

Thank you!

[hallvardkristiansen]

Looks like a lot of the RTP-MIDI messages are too large and there are too many of them going out.
This would surely cause a buffer overflow?

[hallvardkristiansen]

It’s pretty obvious that there are major issues with the messages coming from the computer so I don’t think there was anything wrong on the Teensys in the first place.
Sorry for taking up your time and thank you for trying to help :)

[lathoub]

Thanks for the wireshark dump, very helpful @hallvardkristiansen.

There are indeed a lot of messages being send (see screenshot below, all within 1/10s - from 1 source (10.0.0.6) to 7 targets), but they all look valid (and sending NoteOff as NoteOn with velocity = 0, to preserve running status) and they are all valid. I will check try to use the hex dump and see if the (small W5500) buffer overruns.

The parser can report exceptions, have you enabled them? (see the exceptions example). I'm might see *** ReceivedPacketsDropped messages.

Did you set the largest buffer (2048 bytes) for the W5500 in Ethernet.h? (see the wiki)
#define ETHERNET_LARGE_BUFFERS

[hallvardkristiansen]

Hi, thank you for looking into the logs. I spent some time deeper into the messages last night and it’s difficult to say what might be going wrong.

I previously tried switching to the ethernet3 library and tested various socket numbers, primarily 4, but also 2 and 8, and enabled the exception output and tried to use the raw midi output sketch from examples instead of our code, but the results were the same and no exceptions were triggered as far as we could see. With the ethernet3 lib the teensys froze when they tried to send midi for some reason. But since the issue with receiving from midipipe didn’t improve we didn’t spend anymore time debugging that.

I’m away now and the guys got things working again by simply not sending any messages from midipipe, since the midi from PureData works fine. It’s all quite confusing!