Retain buffer when downgrading bitrate

[jonoward]

[We were planning on submitting this as a PR, as we implemented this in our fork and found it useful in reducing rebuffering; but seems that it may be worth waiting for the refactoring to land. Thought it might be useful just to raise it now]

When changing bitrates, the player currently starts buffering at the new bitrate as soon as it can (based on an estimate on how far to offset the segment index based on the current bandwidth etc). This makes sense for upgrading bitrates, as you want the user to get the better quality bitrate as soon as possible.

But for downgrading bitrate, we may as well let the user watch all of the higher-quality media currently in the buffer, i.e. set the offset at the end of the current buffer. This was probably more noticeable for us as we set a relatively large buffer - 2 minutes - so if the buffer is full, we want to let that media play out.

As a side effect, it reduces the pressure on the buffer, as any bandwidth fluctuations that occur just after a bitrate switch are then less likely to cause a rebuffer, which I think is why we saw an improvement in our stats after adding the behaviour.

[joeyparrish]

Thanks for bringing this up. We're about to publish something similar to this in the next stage of #51. We are no longer going to clear the buffer at all when switching bitrates.

More changes might be needed to implement exactly what you described. Did I understand you correctly that you want switches to higher quality to overwrite the buffer at the playhead, while switches to lower quality content are appended at the end of the buffer?

[jonoward]

Yeah that's pretty much it, when switching to a higher quality we want to switch as soon as possible (so retain just enough buffer in front of the playhead to avoid a rebuffer), but when switching to a lower quality, we may as well play out the buffer (which should be filled with better quality media), so we don't discard any of the buffer, and we append new fragments to the end of the buffer.

[joeyparrish]

The patch we are finishing now will allow something close to this, but not quite. We have a parameter on the video source's switch() method which allows the caller to optionally clear the buffer when switching representations.

One of the outcomes of clearing the buffer is that it naturally requires content to be inserted at video.currentTime, but this results in a noticeable stutter. We would use this flag, for example, when a user manually selects a fixed resolution (disabling ABR), at which point it is acceptable. But for an automated switch, we would obviously want to avoid the stutter.

So it sounds like you would need some way to ask that the content be appended at the current time without clearing the source buffer. SourceBufferManager has two different layers it can query: a map (soon to be list) of inserted segments, and the buffered time ranges of the underlying SourceBuffer. If you were able to say, on switch, that we should forget all (or most) of the content we've already inserted, but not remove it from the SourceBuffer, the result would be that we would start downloading segments that replace those already in the SourceBuffer, but playback wouldn't be interrupted.

[jonoward]

Yes that would work, if we could instruct SBM to forget about all the segments X seconds in front of currentTime, and only apply this rule when upshifting to a higher quality bitrate, that would work for us. Not sure what the value of X should - would probably be ok to be a constant value, e.g. 5-10 seconds, just needs to be a long enough period to download the new initialisation segment and at least one other segment.

[joeyparrish]

You could estimate the best value for X based on the new bitrate & current bandwidth estimate if you wanted to. X = sizeOfSegmentInSeconds * newBandwidthRequirement / currentBandwidthEstimate * fudgeFactorGreaterThanOne. This would be a little larger than the amount of time it would take to download the first segment from the new representation.

[joeyparrish]

v1.4.0 is out now, which no longer clears the buffer by default when switching bitrates. A custom IAbrManager implementation can decide whether or not to clear the buffer via the clearBuffer parameter to IVideoSource.selectVideoTrack().

We do not have an API to clear the buffer starting X seconds ahead, since we do not have an IAbrManager implementation that would make use of it at the moment. That being said, I'd be happy to either add this to our queue to work on for v1.5.0, or accept a pull request for it, whatever you prefer.

Design-wise, perhaps we should have a secondary API along-side selectVideoTrack() like clearBuffer(). Then you could compose the two without making selectVideoTrack more complicated:

videoSource.clearBuffer(video.currentTime + 10, Number.POSITIVE_INFINITY);
videoSource.selectVideoTrack(newTrackId);

[jonoward]

Hi Joey, sorry for the long delay in responding, we're just starting our migration to 1.4 now. I've been looking at your suggestion; do you think it would work for us to leave selectVideoTrack as-is (so no signature change) but also add additional clearVideoBuffer/clearAudioBuffer methods on StreamVideoSource, which we could then use in our custom AbrManager, after we have changed track? I'm happy to submit a PR for the additional methods.

So the pseudo-code in our custom AbrManager would look like:

videoSource_.selectVideoTrack(chosen.id, false); // same as in SimpleAbrManager, buffer not cleared

if (isNewVideoTrackHigherBandwidth) {
this.videoSource_.clearVideoBuffer(video.currentTime + 10, Number.POSITIVE_Infinity);
}

Thanks

[tdrews]

Hi,

My concern with generic clearAudioBuffer and clearVideoBuffer functions are that they would remove/reduce some simplifying assumptions made by Stream, e.g., "there are never any gaps in the buffer".

The safest way to support partial clears is to 1. only allow removing whole segments, and 2. only allow clearing the buffer from the video's current time (plus some offset if desired) to the end of the buffer.

Would something like this work?

/**
 * Clears the audio buffer forward from the segment boundary nearest the video's
 * current time plus |offset| seconds.
 */
VideoSource.clearAudioBuffer(offset) {
  // ...
};

I believe this function could be implemented in Stream without major complications, and would only require the addition of one small function to SourceBufferManager (to remove SegmentRefrences from |inserted_|).

If the "nearest segment boundary" part is too restrictive then I think we could remove this restriction, but it would require a slightly more clever change to SourceBufferManager.

Would something like VideoSource.clearAudioBuffer(offset) work for you? Or do you think you would still need something more generic?

[jonoward]

Yeah I think this will be fine for what we need to do, we don't need to create a gap, we do just want to clear the buffer after a certain time position. Also, aligning to the nearest segment boundary is fine for us. Thanks.

[tdrews]

Great. We're happy to accept a PR for this change :) But we can tackle this if you're short on time.