feature: advanced latency options + handling

[jvxz]

Feature Description

various options for streams that allow more fine grain behavior for dealing with latency. Twitch focus initially; Kick support welcome from contributors.

Expected Behavior

the main option I had behind this is the ability to automatically set (determined) the playback rate of a stream when latency is above a certain (determined) threshold.

example: "if the latency of the current stream is above 5 seconds, set the playback rate to 1.2x." I am aware that Twitch has this built-in already, but I personally find that: the speed it chooses (~1.02x) is too slow, it does not catch up close enough to my liking, & does not trigger when behind enough.

Additional Context

1. Why this feature would be useful

this feature would allow users to catch up more quickly & autonomously. this prevents the manual behavior of having to click the latency reset button every time.

2. Any related ideas or features you think should be considered

when testing to see if this was possible, I discovered that FFZ has some sort of feature to manipulate playback (injected setFFZPlaybackRate symbol on video element). I am currently uncertain how (or where) this feature works, though if interest is expressed in this idea then I would be more than willing to investigate.

If you have an idea of how this feature could be implemented, feel free to suggest it here.

since there is already logic of calculating & tracking latency, it would be a matter of manipulating the stream's video element playbackRate property. a new field in the options of the extension would be in order to allow the user to set the speed to what they desire.

there could potentially be issues when it comes to Twitch manipulating the speed on its own, but that seems like a straightforward prevention.

config example concept:

• Latency (possible new tab to the left):
  • Advanced speed-up: (toggle)
  • Speed-up playback rate: (value, eg: 1.2x)
  • Speed-up threshold: (value, eg: 5 seconds)
  • (...other existing latency options)

[mepherion]

FFZ Enhancing Add-on has this feature if you want to see how they implemented it

[jvxz]

FFZ Enhancing Add-on has this feature if you want to see how they implemented it

thank you for sharing! I had no idea this existed

[Kaedriz]

I have it implemented in simple mode.

Kick only for now. But Twitch shouldn't be too different, it operates on raw DOM node.
Settings don't work yet, too. Need to tune variables in code.

If you are willing to help, appreciated.

[jvxz]

I have it implemented in simple mode.

Kick only for now. But Twitch shouldn't be too different, it operates on raw DOM node. Settings don't work yet, too. Need to tune variables in code.

If you are willing to help, appreciated.

@Kaedriz thanks so much! I will try my best to work on the Twitch side today, if not then as soon as I have the time

[Kaedriz]

I have done some work on it.

Can you test it if it works reliably?

I'm trying to figure out if I need to explicitly disable twitch native one, so they don't interfere, but it appears to work.

Also, it only works on Low Latency mode I think, as in without it, it should play faster, but it doesn't budge much.

I had some digging and without it, <video> player doesn't expose playbackRate so changing it simply fails, you can check on to that if you want.

For now, I will try to integrate this with settings and test things out.

[jvxz]

this doesn't seem to be working for me. I think the issue is that the twitchUtils().getMediaPlayerInstance() util doesn't actually expose the raw video element currently, and that is needed to reliably set the playback rate

I have actually been writing my own implementation for the past couple hours. I made a dedicated settings tab for latency-related options, and I extended the twitchUtils().getMediaPlayerInstance() util to expose the raw video element via renderSurface.video.element(). I can push it to the PR if you would like to try it out. it seems to be reliable from my testing

[jvxz]

Also, it only works on Low Latency mode I think, as in without it, it should play faster, but it doesn't budge much.

testing with this disabled, you seem to be right about this. it seems as if the player resets itself when it goes below 5 or 6 seconds when low latency is disabled

a solution that comes to mind right now is to inform the user that, if they select a threshold <6/5, then a warning will be shown that low latency is required to be enabled, or else the player will reset repeatedly. that, or logic can be written to detect if the user has low latency enabled and a hard cap can be applied to the threshold input in settings including the aforementioned warning

[jvxz]

I have actually been writing my own implementation for the past couple hours

if you would like to see some changes I made you can check it out here https://github.com/jvxz/enhancer/tree/latency-reducer

[Kaedriz]

Yea, but with Low Latency off, even direct manipulation of <video> doesn't work for me.

It just revert itself.

[jvxz]

really strange. I have success in my implementation of setting the playback rate with low latency disabled via the codebase, but not in devtools like you demonstrated

[Kaedriz]

I have seen that you choose separate module route.

I will try it in practice tomorrow, and detach kick latency module to separate one.

You can maybe place settings closer together to stream latency one in file for readability.

[Kaedriz]

Also, you can probably remove my attempt on twitch, so it wouldn't interfere with yours.