MusicXML- No support for interweaving of staves in MusicXMLImporter

[mxAlexG]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi,
This issue occurs with multiple files. This screenshot shows what happens.

Musescore Web Player:
https://user-images.githubusercontent.com/105655048/185884706-68b45825-9f70-4790-9238-a3cf1d09153e.mp4
AlphaTab:
https://user-images.githubusercontent.com/105655048/185884641-7845d19e-02f4-4027-8919-7582c31d7c03.mp4

Expected Behavior

Similar to Musescore player's, or as part of the treble clef notes where they should- less appealing though.
Support for interweaving of staves.

Steps To Reproduce

None.

Link to jsFiddle, CodePen, Project

No response

Found in Version

1.2

Platform

Web

Environment

- **Linux, Ubuntu**:
- **Brave, others**:

Anything else?

No response

[Danielku15]

Can you try to describe this bug in a more minimal reproducible example. I cannot grasp from this screenshot and explanation what exact problem you are facing. Some more formal description, example files/fiddles etc. will help me to actually analyze what problem you are facing and what might cause it.

[mxAlexG]

Hi,
Ignore everything and only look at what's marked with red.

[Danielku15]

Looks like one of the biggest pain points in the MusicXML spec. MusicXML across applications has:

1. A terrible inconsistency on how to assign notes semantically correct to staves and voices. Every application writes things a bit differently interweaved and the order in the file is relevant for assigning them correctly.
2. There is also a very weird feature where instead of properly filling a measure/bar with rests and notes, you can just annotate notes to be shifted back or forth in time. But not only in time, but also just in position sometimes elements are visually shifted to look like they are at a different place. alphaTab doesn't support this weird shifting but requires bars/measures to be filled properly with content (e.g. rests) to align.

If you can provide a file holding exactly this measure from the screenshot I could check from where in your case this misalignment and assignment of notes comes from. Likely it is a mix of the upper two points where point 1 could be a bug or unhandled edge-case and for point 2 it is not supported.

[mxAlexG]

Hi,
Thanks for the response.
I think this frequently happens when using advanced pieces downloaded form Musescore. I open these files with Musescore (desktop) and Soundslice. Both render them properly. But, if you look at how everything is "designed"/ laid out in Musescore editor you'd notice a bit of a mess. I would understand that alphaTab can't do everything perfectly- at least yet, but I think most renderers out there base their importers on Musescore's exporter since I think it's the most popular source for pieces with the format.

What I thought initially was that this behavior occurs because of how you implemented the logic for parsing multi staff tracks. I assumed that you implemented it in a way where any note below C4 gets automatically added to the second staff, but because the note is also beamed with other notes, they also get added to the second staff. I tried to look into the importer's code but there's a lot of intertwined code that requires me to have a knowledge of the entire codebase, which I don't think is something I can do within reasonable time limits.
This is file I used in the example- bar 3:
lg-8102429.zip

[Danielku15]

Note: some parts below are more personal notes for myself to remember in future. You don't need to understand everything but depending on how deep you are in this topic already it might give you some interesting insights 😉

Looking quickly through the XML the root cause is likely the fact that we do not support at this point the interweaving of staffs and adapting the time axis with backup tags.

The Music XML contains first the rest of staff 1, then proceeds to add notes to staff 2 advancing on the time axis. Then it continues with the last 4 notes of staff 1. And then it jumps back on the time axis with the backup tag, adding the second voice of staff 2 from the beginning.

Visually the order of the tags in the file are like:

This is exactly the semantics I meant with point 2 above. Staff 1 is actually not "properly" filled with rests but jumping into the future from tag-1 to tag-13 by the duration caused by the notes on staff 2.
And via the backup it jumps back on the time axis to reset the positioning of the notes for the second voice.

This is quite tricky to handle because you never actually know if from the flow in the file, some previously caused gaps are again filled. It would be possible to again make a backup after tag 25 and start filling additional notes into staff 1. This is a very visually oriented way of writing music notation instead of properly describing semantically for each staff the notes contained like alphaTab expects it.

One hacky solution for this praticular file could be to assume that after a backup an already filled voice of a staff is not filled filled again with notes inbetween. Then I could fill up staff 1 with rests caused by the notes 2-12 and by this align the time axis of staff 1 and the first voice of staff 2.

To solve this problem technically, alphaTab would need to support notes to be "floating" somewhere on the time axis, without having a strict rule that every element in a voice starts there where the previous one ends and then position items accordingly.

This means the MusicXML would need to be adjusted in the following way:

Change the MusicXML parser to follow a time axis while parsing a measure which advances with every Note element parsed according to the duration tag. Independent of the existing elements in a staff and voice. Beats are placed at the current time position. backup and forward tags adjust the time axis.

From a data model perspective this means we need a way to allow gaps between the notation elements.
e.g. the first beat in a voice starts at tick 0 and takes 960 ticks (quarter note). the second beat starts at 1920 and takes also 960 ticks leaving us with 960 ticks of "nothing" between the two beats. I see two possibilities to tackle this:

a) We 'simply' allow gaps. This means we need to ensure across the whole codebase to treat the Beat.displayStart and Beat.playbackStart with this new rule. Layouting and Midi generation would need to accept this gaps correctly. On the MusicXML parser we might have to "squeeze" beats in-between two other beats if the cursor requires this. We also need to assume then that files are really properly aligned timely because theoretically overlaps of beats in one voice would be possible then.
Pros: This gives us quite some flexibility in layouting.
Cons: It could interfere easily with other features where we consolidate the time axis which are already fragile. (I'm looking at you Grace Beats 👈👀 ). And IMO it is unclean to just have gaps without properly timed rests in the model.

b) We store this additional gap as an auxiliary information on the previous beat and respect this gap across the codebase when layouting and generating midi.
Pros: It also gives us some additional flexibility in layouting, but we could easier ensure some consistency rules and avoid overlaps (requiring gaps to be >= 0)
Cons: Like on option a.

c) Similar to option b. We remember the information about the "gaps", but we do this in the MusicXML parser. When consolidating the model, we fill the gaps with empty or rest beats.
Pros: The specific behaviors of MusicXML do not spoil overall codebase with this new rule. The existing algorithms in layouting and midi generation would work.
Cons: It is not guaranteed that those gaps can be filled with clean rests. e.g. it could be 80% of a quarter note not having a proper symbolic. The music sheet might get "noisy" (loaded with glyphs) unless we make those rests invisible. So we might need a concept of "gap beats" having no display and a duration specified in midi ticks instead of the usual note steps (8th, quarter, whole) and dots.

[mxAlexG]

Hi,
Thanks for the response.
This information could prove useful. I think we could say that these issues- at least for now- can then be considered as unrelated to alphaTab and should be handled independently.

[Danielku15]

I would say they still relate to alphaTab, it is just that the alphaTab cannot handle such interweaved MusicXML files at this point. It also relates to MusicXML not having a proper specification on how such scenarios should be handled when interpreting the file leaving it up to the software authors to find out what to write and read in such cases.

I tested various tools and software and many behaved slightly different than MuseScore but also various tools worked as expected.