Chromecast player stalling when gap in live manifest

[maxi297]

Have you read the FAQ and checked for duplicate open issues?: Yes

What version of Shaka Player are you using?: v2.4.4

Can you reproduce the issue with our latest release version?: Yes

Can you reproduce the issue with the latest code from master?: Yes

Are you using the demo app or your own custom app?: Both

If custom app, can you reproduce the issue using our demo app?: Yes

What browser and OS are you using?:
Chromecast (for example: V1, V2, Ultra)

What are the manifest and license server URIs?:
Our manifest contains SegmentTemplate with a gap like this:

             <SegmentTemplate timescale="10000000" media="S!dxIDVP7...8BFgZnDU9QUEUtV0lERVZJTkV4IOIBFQGv/QualityLevels($Bandwidth$)/Fragments(video=$Time$)" initialization="S!dxIDVP7...8BFgZnDU9QUEUtV0lERVZJTkV4IOIBFQGv/QualityLevels($Bandwidth$)/Fragments(video=Init)">
                <SegmentTimeline>
                    <S t="15440366781137770" d="30000000" r="12"/>
                    <S d="20000000"/>
                    <S t="15440367201137770" d="30000000"/>
                </SegmentTimeline>
            </SegmentTemplate>

Our stream is DRM protected which will make it hard to reproduce. We could provide the URIs in private and more information if needed.

What did you do?
We have played a live stream with gaps in the manifest on Chromecast. When the gap occurs, the player buffers for many seconds. We do not own the manifest and have to deal with those gaps.

What did you expect to happen?
There is a jump over the gap and continue.

What actually happened?
The player got stalled until it was outside of the availability window.

Do you have any idea why this happens?
1- In GapJumpingController.onPollGapJump_, We successfully jumped over the gap. We then exit GapJumpingController.onPollGapJump_
2- We return in GapJumpingController.onPollGapJump_ and enter in handleStall_. Conditions are met (we haven't understood what this implies entirely) so that the player prints 'Flushing media pipeline due to stall inside buffered range'
3- The player is stalled until it was outside of the availability window.
4- The player returns in the seek range and can play again until the next gap.

Workaround for now
We are not completely clear about why the Chromecast is stalling. We believe that this device is not fast enough to seek. We can see that this problem does not occur on web with a Chrome browser.

We have implemented a workaround and are eager to see what you think of it. Our workaround is to add a grace period after a jump before stalling. Basically, everytime we jump, we keep the timestamp of NOW and before stalling, we check if we have jumped recently over gap. Here is the interesting part:

// shaka.media.GapJumpingController.prototype.onPollGapJump_ ...
  if (gapIndex == -1) {
    if (this.gracePeriodIsPast_(currentTime)) {
      this.handleStall_();
      return;
    } else {
      shaka.log.info('Stalling within grace period');
      return;
    }
// ...
  if (isGapSmall || jumpLargeGap) {
// ...
    this.lastJumpPerformedTime_ = this.getNow_();
  }

/**
 * @param {number} currentTime
 * @return {boolean} if the grace period is past
 */
shaka.media.GapJumpingController.prototype.gracePeriodIsPast_ =
 function(currentTime) {
  return this.lastJumpPerformedTime_ +
    this.config_.gapJumpingGracePeriodInSeconds
    <= this.getNow_();
};

/**
 * @return {number} now
 */
shaka.media.GapJumpingController.prototype.getNow_ = function() {
  return new Date().getTime() / 1000;
};

EDIT: The gaps on the live streams occur only once every 22m30s. Therefore, we do not fear to have many gaps in a short period of time. This fix would not work in that case because we might jump over the first gap, might end up near another gap and our gapJumpingGracePeriodInSeconds would not allow the player to jump again. As of now, we are trying to gather more data with a gapJumpingGracePeriodInSeconds = 3. We might revisit this value if we see any side effects.

Questions:
1- Have you already experienced this issue? If so, how did you dealt with?
2- Do you have any comment/solution for our workaround?

[theodab]

We've definitely had problems with slow hardware problems with Chromecasts before. See issue #1411, for example. So your assumption about the cause of this seems reasonable, though without being able to reproduce the issue I can't say for certain.

Looking at the code, it also seems possible that our current check for stalling inside a buffered range is too aggressive. It assumes that if the media is "playing" but has had a playback rate of 0 for more than a second straight, the video has stalled. Maybe on slow hardware, like on Chromecasts, it simply sometimes takes more than one second to seek sometimes, so we are assuming the stream has stalled prematurely?

You can send details on reproduction to shaka-player-issues@google.com.

[theodab]

To be clear, there are some ideas we could try, but we will need to be able to test them to be sure they actually solve the problem.

[maxi297]

I have sent you the information by email. If anything else is need, tell us and we will provide the information as soon as possible!

[joeyparrish]

We received the info by email. Thank you!

[theodab]

Ok, I was able to reproduce the problem with a (short) custom manifest. This change solves the problem on that custom manifest; I haven't tested the fix on your sample. I'll try it later, or you can try it yourself if you want. I won't close the issue until I know it's an actual fix.

[alexgerv]

Hi @theodab, the changes on e5b2428 are helping, but we actually observed delays greater than 0.5 seconds for the player to recover from a gap, more in the 1 or 2 seconds range, sometimes even more on a Chromecast V1 or embedded. Ideally, this value would be configurable. Any thoughts ?

[theodab]

We plan on gathering more thorough data on platform-specific gap behavior in the future. Hopefully that will help us better account for V1 Chromecasts, and other platforms with weak hardware.
I can write a CL for a configurable gap threshold, in the meantime, and see what the rest of the team thinks. That'd be straightforward enough.

[TheModMaker]

That number specifies the distance from a gap where we will jump it. So with that change, we'll jump a gap once we are 0.5 seconds from it. This ensures we seek before the browser stalls due to not enough media. This value has no effect on the stall detection. What is probably happening is the Chromecast is taking too long to start playing after the gap jump so the stall detection kicks in and we do another seek. The delay before we handle a stall would be better to be configurable.

[alexgerv]

It probably does not hurt to attempt the seek a bit earlier, but that's correct, we observed that the real issue was the stall detection kicking in too fast, before the chromecast even had the chance to seek and resume.
Detecting the stall is under the gapJumpTimer .25 seconds loop, but it can also be short-circuited by a segment being appended.
The "grace period" logic proposed by @maxi297 really did alleviate the issue.

[joeyparrish]

Work so far cherry-picked for v2.4.6.

[vaage]

@alexgerv I am sorry that progress on this issue has been dormant for so long.

I noticed that you mentioned in your last comment that stall detection is kicking too fast. As part of Issue #1839 we are making the stall detection logic configurable. Would that help you make progress until we find time to investigate gaps and stalls more carefully?

In offline converstaions, @joeyparrish and I have talked about how buffering, gap detection, and stall detection may need to be more cooperative, which would be in-line with your idea of grace periods. I'm confident that we'll find a solution that will work for you and others, but we will need time.

What may help this issue get more attention and slotted for a milestone is if you were to summarize what behaviors you need and open an enhancement request. Enhancement requests are easier for community members to rally behind, increasing their priority, and increasing the odds that they'll get resources directed to them.