WebvttParser creates duplicate CuesWithTiming when handling cues sharing same start/end timestamps

[JunkFood02]

Version

Media3 1.3.0

More version details

media3-extractor

Devices that reproduce the issue

Android Studio

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

Use WebvttParser.parse() to parse the vtt captions below:

private const val vttSample = """WEBVTT
Kind: captions
Language: en

00:00:00.000 --> 00:00:01.712
[MUSIC PLAYING]

00:00:02.533 --> 00:00:03.950
SPEAKER: As an
Android engineer, I

00:00:03.950 --> 00:00:06.350
have seen firsthand how
the ecosystem is growing--
"""

val cuesWithTimingList = buildList {
            WebvttParser().parse(
                vttSample.toByteArray(),
                SubtitleParser.OutputOptions.allCues()
            ) {
                add(it)
            }
        }

Expected result

The cuesWithTimingList contains 3 items or lines

Actual result

The list contains 4 items, see screenshot

Media

WEBVTT
Kind: captions
Language: en

00:00:00.000 --> 00:00:01.712
[MUSIC PLAYING]

00:00:02.533 --> 00:00:03.950
SPEAKER: As an
Android engineer, I

00:00:03.950 --> 00:00:06.350
have seen firsthand how
the ecosystem is growing--

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[icbaker]

The duplication is expected for cues that actually overlap, that's checked in a test case here:

media/libraries/extractor/src/test/java/androidx/media3/extractor/text/webvtt/WebvttParserTest.java

Line 338 in 12aa637

public void parseWithOverlappingTimestamps() throws Exception {

This duplication is currently required due to the way we implement WebVTT's simultaneous cues handling (by creating synthetic CuesWithTiming objects representing the overlapping state) - which is mostly done inside WebvttSubtitle:

media/libraries/extractor/src/main/java/androidx/media3/extractor/text/webvtt/WebvttSubtitle.java

Line 81 in 12aa637

// Steps 4 - 10 of https://www.w3.org/TR/webvtt1/#cue-computed-line

If we didn't create these synthetic cues here, we would have lost too much information to correctly implement the layout rules later in the playback pipeline. It's still not perfect (see e.g. google/ExoPlayer#10980), and it would be better to implement this layout logic later, but it's not likely something we're going to change soon.

However, this duplication is not expected for consecutive cues where one cue ends at the same time the next cue starts. I can reproduce that the duplication does occur in this case, I'll look into fixing it.

[JunkFood02]

Thanks for the quick and detailed response! Edited the title since I didn't intend to work with subtitles with overlapping timestamp

[szaboa]

@icbaker as I understood this was just a preparation to handle overlapping vtt cues?

If we didn't create these synthetic cues here, we would have lost too much information to correctly implement the layout rules later in the playback pipeline. It's still not perfect (see e.g. google/ExoPlayer#10980), and it would be better to implement this layout logic later, but it's not likely something we're going to change soon.

Do you have suggestions on the "layout logic"?

[icbaker]

@szaboa I'm not sure I completely understand the question - do you mind filing a new question issue with a bit more detail? (I'm likely to lose track of this closed one)