SSA/ASS subtitles - Overlapping start/end times and position tag is not handled

[XBigTK13X]

Use case description

When playing videos files with SSA/ASS subtitle tracks, styled subtitles are mashed together with conversation. For example, many animated Japanese television shows have subtitles that float on screen to translate things that aren't part of speech. In ExoPlayer, this results in the subtitles flipping back and forth between conversation and the translated symbols. This leads to neither being on screen long enough to be read.

Proposed solution

Integrate libass as a supported subtitle renderer

Alternatives considered

The only alternative would be rolling a new renderer from scratch for ExoPlayer. All major media players are currently using this library. I did look through other issues on this project about SSA/ASS subtitles, but none of them lead to a request for integrating this library as a solution.

[tonihei]

I had a quick look at the library and from what I see it produces overlay images including the subtitles. Is that correct? I'm afraid that's not that easy to support in ExoPlayer, because you'd need some form of special renderer that draws the overlay. See #5860 for details around writing such a renderer. It would also require an extra surface and at least some extra layer in the UI module to actually draw the overlay.

Depending on how it looks like, we could probably accept pull requests for an extension that integrates this library and provides the necessary integration layers. But we should definitely keep the basic text-only subtitle extraction for SSA/ASS we currently have. The problem you are describing sounds more like a bug in our implementation and it would be great if you can provide example media and some guidance on the expected result, so that we can fix our support for the text-only version.

[XBigTK13X]

Thank you for the feedback. That points me in the right direction. I will try adding an alternative SSA renderer locally to the ExoPlayer source to see if that solves the problem for my project.

Here is a snippet from a subtitle file that renders incorrectly in the current ExoPlayer.

[Script Info]
; Script generated by Aegisub 3.2.2
; http://www.aegisub.org/
Title: ExportedSubs
ScriptType: v4.00+
WrapStyle: 0
PlayResX: 1280
PlayResY: 720
ScaledBorderAndShadow: yes
YCbCr Matrix: TV.709

[Aegisub Project Garbage]
Last Style Storage: Aikatsu 1
Audio File: yuru05_premux.mkv
Video File: yuru05_premux.mkv
Keyframes File: yuru05_premux_keyframes.log
Video AR Mode: 4
Video AR Value: 1.777778
Video Zoom Percent: 0.875000
Scroll Position: 1422
Active Line: 1437
Video Position: 33970

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,Cabin,51,&H00FAFAFA,&H000019FF,&H003F1613,&HC42F100E,-1,0,0,0,100,100,0,0,1,2.4,1.1,2,150,150,40,1
Style: Default it,Cabin,51,&H00FAFAFA,&H000019FF,&H003F1613,&HC42F100E,-1,-1,0,0,100,100,0,0,1,2.4,1.1,2,150,150,40,1
Style: Default-alt,Cabin,51,&H00FAFAFA,&H00002EFF,&H00201843,&HC4171232,-1,0,0,0,100,100,0,0,1,2.4,1.1,2,150,150,40,1
Style: Signs,Lato,54,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,0,0,5,0,0,0,1
Style: OP,FOT Kafu Nagomi Std Stripped,40,&H00FFFFFF,&H000000FF,&H00BA5B2F,&H00000000,0,0,0,0,100,100,0,0,1,1,0,2,10,10,25,1
Style: OP jp,FOT Kafu Nagomi Std Stripped,40,&H00FFFFFF,&H000000FF,&H00BA5B2F,&H00000000,0,0,0,0,100,100,0,0,1,1,0,8,10,10,25,1
Style: ED,Ruzicka TypeK,40,&H00FFFFFF,&H000000FF,&H00717171,&H00000000,-1,0,0,0,100,100,0,0,1,2.1,0,7,40,10,25,1
Style: ED jp,Ruzicka TypeK,40,&H00FFFFFF,&H000000FF,&H00717171,&H00000000,-1,0,0,0,100,100,0,0,1,2.1,0,7,40,10,71,1

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 10,0:09:18.07,0:09:21.13,Default it,,0,0,0,,I'll start with something simple.
Dialogue: 0,0:09:21.13,0:09:30.39,Signs,,0,0,0,,{\clip()\b1\fs14.5\move(899,300,899,339,28,9245)\frz4.3\fax0.175\c&H6054FF&\blur0.6}Mega Tasty Milk
Dialogue: 10,0:09:21.71,0:09:26.30,Default it,,0,0,0,,Tonight's meal is a pasta soup that can be made with a single camping pot.
Dialogue: 10,0:09:26.30,0:09:30.39,Default it,,0,0,0,,It leaves no extra broth, making it ideal for camping and hiking.
Dialogue: 0,0:09:30.39,0:09:30.89,Signs,,0,0,0,,{\pos(191.43,146.14)\fnBurst My Bubble\b1\frz29.42\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:30.39,0:09:30.89,Signs,,0,0,0,,{\pos(494,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz345.8}Bacon
Dialogue: 0,0:09:30.39,0:09:30.89,Signs,,0,0,0,,{\pos(818,62)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz343.8}Asparagus
Dialogue: 0,0:09:30.39,0:09:30.89,Signs,,0,0,0,,{\pos(1020,112)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz342}Mushrooms
Dialogue: 0,0:09:30.89,0:09:31.39,Signs,,0,0,0,,{\pos(225.716,131.283)\fnBurst My Bubble\b1\frz21\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:30.89,0:09:31.39,Signs,,0,0,0,,{\pos(479,60.33)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz357}Bacon
Dialogue: 0,0:09:30.89,0:09:31.39,Signs,,0,0,0,,{\pos(792,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz354}Asparagus
Dialogue: 0,0:09:30.89,0:09:31.39,Signs,,0,0,0,,{\pos(1039,121)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz332.5}Mushrooms
Dialogue: 10,0:09:31.05,0:09:33.90,Default it,,0,0,0,,Take your previously chopped ingredients,
Dialogue: 0,0:09:31.39,0:09:31.89,Signs,,0,0,0,,{\pos(191.43,146.14)\fnBurst My Bubble\b1\frz29.42\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:31.39,0:09:31.89,Signs,,0,0,0,,{\pos(494,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz345.8}Bacon
Dialogue: 0,0:09:31.39,0:09:31.89,Signs,,0,0,0,,{\pos(818,62)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz343.8}Asparagus
Dialogue: 0,0:09:31.39,0:09:31.89,Signs,,0,0,0,,{\pos(1020,112)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz342}Mushrooms
Dialogue: 0,0:09:31.89,0:09:32.39,Signs,,0,0,0,,{\pos(225.716,131.283)\fnBurst My Bubble\b1\frz21\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:31.89,0:09:32.39,Signs,,0,0,0,,{\pos(479,60.33)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz357}Bacon
Dialogue: 0,0:09:31.89,0:09:32.39,Signs,,0,0,0,,{\pos(792,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz354}Asparagus
Dialogue: 0,0:09:31.89,0:09:32.39,Signs,,0,0,0,,{\pos(1039,121)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz332.5}Mushrooms
Dialogue: 0,0:09:32.39,0:09:32.89,Signs,,0,0,0,,{\pos(191.43,146.14)\fnBurst My Bubble\b1\frz29.42\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:32.39,0:09:32.89,Signs,,0,0,0,,{\pos(494,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz345.8}Bacon
Dialogue: 0,0:09:32.39,0:09:32.89,Signs,,0,0,0,,{\pos(818,62)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz343.8}Asparagus
Dialogue: 0,0:09:32.39,0:09:32.89,Signs,,0,0,0,,{\pos(1020,112)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz342}Mushrooms
Dialogue: 0,0:09:32.89,0:09:33.39,Signs,,0,0,0,,{\pos(225.716,131.283)\fnBurst My Bubble\b1\frz21\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:32.89,0:09:33.39,Signs,,0,0,0,,{\pos(479,60.33)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz357}Bacon
Dialogue: 0,0:09:32.89,0:09:33.39,Signs,,0,0,0,,{\pos(792,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz354}Asparagus
Dialogue: 0,0:09:32.89,0:09:33.39,Signs,,0,0,0,,{\pos(1039,121)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz332.5}Mushrooms
Dialogue: 0,0:09:33.39,0:09:33.90,Signs,,0,0,0,,{\pos(191.43,146.14)\fnBurst My Bubble\b1\frz29.42\fs40\c&H040303&\blur0.65}Onions
Dialogue: 0,0:09:33.39,0:09:33.90,Signs,,0,0,0,,{\pos(494,58)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz345.8}Bacon
Dialogue: 0,0:09:33.39,0:09:33.90,Signs,,0,0,0,,{\pos(818,62)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz343.8}Asparagus
Dialogue: 0,0:09:33.39,0:09:33.90,Signs,,0,0,0,,{\pos(1020,112)\fnBurst My Bubble\b1\fs40\c&H040303&\blur0.65\frz342}Mushrooms
Dialogue: 10,0:09:33.90,0:09:37.49,Default it,,0,0,0,,and fry them in olive oil and chopped garlic.
Dialogue: 10,0:09:37.90,0:09:41.28,Default it,,0,0,0,,Mix in one hundred and fifty milliliters of water,
Dialogue: 10,0:09:41.65,0:09:45.41,Default it,,0,0,0,,add one cube of consomme, and bring to a boil.

When drawn through a player using libass, it looks like this.

Here is a clip showing the undesirable rendering behavior.

Now that I look at the subtitle data, I wonder if a second rendering track at the top for Signs would be sufficient to fix the dialogue getting replaced.

[tonihei]

The particular reasons why this goes wrong here are:

1. We don't expect overlapping subtitle times at the moment. We should detect that and and return all active subtitles for a given time so that they can all be displayed simultaneously.
2. We don't parse the position tag of the subtitles to place it at the right position.

Step 1 is more important to remove the strange flickering of subtitles. Step 2 is nice to have because it would actually render the text at the right position.