synthesize() clips notes when they overlap in time and pitch

[justinsalamon]

If the MIDI file contains two notes (in the same instrument) that have the same pitch and overlap in time, the second of the two will not get synthesized. I.e., say I have a C4 from 1-3s, and another C4 from 2.5-4s: the second note gets "killed" during synthesis.

Admittedly two overlapping notes by the same instrument is not physically possible for all instruments (it is for a guitar for example though), but I think that the correct behavior should be to give the onset of the second note priority over the offset of the first note?

[craffel]

I think what I did to resolve this was to create a MIDI file with this behavior (overlapping same-note-same-instrument notes) and played it back in a few DAWs, and found that the common behavior was that when there was a note off to kill all notes with that instrument and pitch, so that's the behavior I replicate here. Personally, I wouldn't commit to saying LIFO or FIFO is correct here, so my inclination is just to kill both because MIDI data has no built-in way to indicate precedence. If there's a MIDI spec somewhere indicating what the "standard" way to do things is though, or your experience is different with different DAWs, I would be happy with making the change.

[justinsalamon]

tbh I haven't surveyed many DAWs in this regard, and you're probably right that there's no established convention. I've been working primarily with Logic Pro, where if two midi notes overlap in time/pitch they both just get synthesized (both the offset of the current note and the onset of the new note), as if they were just notes with different pitches. I found this functionality very useful, since being aware of every note onset was of the utmost importance when creating manual annotations. It also so happens that some of the midi files I got from you already contained this type of overlap, and if it weren't for this functionality in Logic I may have missed the presence of these notes altogether.

This is what they say in the Nyquist docs:

Be aware that overlapping notes on the same pitch can be a problem for some synthesizers. The following example illustrates this potential problem:

!TEMPO 60
C Q #160   * starts at time 0,   ends at 1.6 sec
D I        * starts at time 1,   ends at 1.8 sec
C Q        * starts at time 1.5, ends at 3.1 sec?

At one beat per second (tempo 60), these three notes will start at times 0, 1, and 1.5 seconds, respectively. Since these notes have an articulation of 160, each will be on 160% of its nominal duration, so the first note (C) will remain on until 1.6 seconds. But the third note (another C) will start at time 1.5 seconds. Thus, the second C will be started before the first one ends. Depending on the synthesizer, this may cancel the first C or play a second C in unison. In either case, a note-off message will be sent at time 1.6 seconds. If this cancels the second C, its actual duration will be 0.1 rather than 1.6 seconds as intended. A final note-off will be sent at time 3.1 seconds.

Despite the lack of convention, my intuition says that the synthesis should match the concept of "note" as closely as possible, i.e. if there's a note between 1-3s and another between 2-4s, both should be audible, despite the overlap. The fact that the second note might get cancelled due to the note-off message of the first note feels more like an artifact of the midi protocol than a desired functionality. But again, that's just my personal preference.

EDIT: the occurrence of overlapping notes with the same pitch is perfectly plausible musically (in most string instruments), so I don't see why it should be considered a "corner" or "erroneous" case other than an artefact of the limitations in the design of the midi protocol.

[craffel]

So, I just reviewed the code for synthesize, and it's not synthesize which has this behavior - it builds up note waveforms independently and combines them additively. For example,

In [1]: import pretty_midi

In [2]: pm = pretty_midi.PrettyMIDI()

In [3]: pm.instruments.append(pretty_midi.Instrument(0, 0))

In [4]: pm.instruments[0].notes = [pretty_midi.Note(100, 60, .1, 1.3), pretty_midi.Note(100, 60, 1.1, 4.5)]

In [5]: a = pm.synthesize(fs=44100)

Listening to a, the second note certainly isn't getting cut off at 1.3 seconds, it rings until 4.5 seconds. I was confused because in the fluidsynth method, the second note will get cut off, but handling all of this is 100% the prerogative of the fluidsynth program, not pretty_midi. We just pass messages to fluidsynth and let it do what it wants with them.

Now, if were to load in this data from a MIDI file instead of constructing it in pretty_midi, when pretty_midi loads in the notes it will kill both at 1.3 seconds, because of this: https://github.com/craffel/pretty-midi/blob/master/pretty_midi/pretty_midi.py#L268 but again, in pure-MIDI land you have to make an explicit choice of either LIFO or FIFO - there's no way to associate a particular note off with a particular note on in MIDI (though there is in pretty_midi, of course!). I think you are basically advocating that it should be FIFO (which is apparently what Logic must be doing?), which I could maybe get behind, but it would complicate the loading code a bit. In the ideal case we could let the user decide what their desired behavior was.

if there's a note between 1-3s and another between 2-4s, both should be audible, despite the overlap.

They are both audible, one just gets cut off at 3s instead of 4s.

The fact that the second note might get cancelled due to the note-off message of the first note feels more like an artifact of the midi protocol than a desired functionality.

It is certainly an artifact of the MIDI protocol. To me, the "desired functionality" is the convention, which I had sort of concluded was "kill all notes on this channel/pitch when a note off is received". At the very least, it could be a bit confusing if someone read in a MIDI file with FIFO behavior, and then used fluidsynth and got the kill-everything behavior.

[justinsalamon]

I think you are basically advocating that it should be FIFO (which is apparently what Logic must be doing?), which I could maybe get behind, but it would complicate the loading code a bit. In the ideal case we could let the user decide what their desired behavior was.

I guess I am, at least for synthesize(). But think letting the user decide via an optional parameter might indeed be the best option here.

To me, the "desired functionality" is the convention, which I had sort of concluded was "kill all notes on this channel/pitch when a note off is received"

I'm not entirely sure you can call it a convention if Logic (which is not as popular as e.g. Pro Tools but still one of the biggies) doesn't follow it. From the Nyquist docs it also seems like DAWs are divided on this. In the absence of a convention, I think it makes sense to choose one option as the default but support the alternative too.

In any case, this is a relatively minor issue, but I know you'd make at least one person happy if you supported FIFO synthesis too :)

[craffel]

I guess I am, at least for synthesize(). But think letting the user decide via an optional parameter might indeed be the best option here.

Just to be clear, synthesize does not have this issue, it's caused by how pretty_midi loads in a MIDI file and converts it to its internal representation. If you hand-construct a PrettyMIDI instance without loading a MIDI file as I did in my last comment, you won't have this issue.

I'm not entirely sure you can call it a convention if Logic (which is not as popular as e.g. Pro Tools but still one of the biggies) doesn't follow it. From the Nyquist docs it also seems like DAWs are divided on this. In the absence of a convention, I think it makes sense to choose one option as the default but support the alternative too.

I think the ideal solution, because there is indeed a lack of convention, is to allow the user to decide. This would take a good bit of work though, I think, but I will leave this issue open as a reminder when I have some time to work on it.

[justinsalamon]

Just to be clear, synthesize does not have this issue, it's caused by how pretty_midi loads in a MIDI file and converts it to its internal representation. If you hand-construct a PrettyMIDI instance without loading a MIDI file as I did in my last comment, you won't have this issue.

Sorry, yes. Doesn't this mean that creating a midi file from scratch with pretty_midi, saving it to disk, and then loading it back with pretty_midi will give different results if you try to synthesize before/after saving/loading?

I think the ideal solution, because there is indeed a lack of convention, is to allow the user to decide. This would take a good bit of work though, I think, but I will leave this issue open as a reminder when I have some time to work on it.

👍

[craffel]

Sorry, yes. Doesn't this mean that creating a midi file from scratch with pretty_midi, saving it to disk, and then loading it back with pretty_midi will give different results if you try to synthesize before/after saving/loading?

Yes, but there's no way to avoid this in general. pretty_midi's representation of a score is slightly different, and in this way better-specified, than MIDI.