Input lag with vsync enabled due to limited polling rates

[alice-i-cecile]

Problem

@aevyrie has observed noticeable input lag in Bevy applications when vsync is enabled.

The most immediate source of this is quite obvious: we're only fetching the input state at the start of each frame, but are rendering is done at the end of the frame.

Input events appear to be moved into the Bevy app by the winit runner:

bevy/crates/bevy_winit/src/lib.rs

Line 221 in de8edd3

pub fn winit_runner_with(mut app: App) {

At 60 fps, this means 16 ms of lag, which is noticeable for some applications: namely for precise cursor movement (FPS, GUI applications) and rhythm games.

Possible solutions

• Poll more regularly (somewhat limited by the OS I believe)
  • I suspect this could be done at the end of each stage
  • Perhaps a dedicated input-polling thread architecture would help?
• Poll closer to rendering time

Either solution will involve some trickiness, as we must pierce the Bevy schedule in some fashion in order to insert fresh input events into the World at the right time, rather than merely at the beginning of each pass over the schedule.

[aevyrie]

IIRC, we are limited by the winit event loop, and we can't poll mouse input asynchronously.

I had some ideas that I could prototype, that effectively act as a frame-limiter, which is related to #1343.

How bevy currently behaves with vsync on:

0ms -----------------------
        Start of event loop. Get input. Do stuff.
4ms     Done.
        --
        Sit around and do nothing
        --
        Send to GPU, present frame (input is 16ms out of date)
16ms ----------------------

How bevy currently behaves with vsync off:

0ms -----------------------
        Start of event loop. Get input. Do stuff.
        Send to GPU, present frame (input is 4ms out of date)
4ms     Done.
        Start of event loop. Get input. Do stuff.
        Send to GPU, present frame (input is 4ms out of date)
8ms     Done.
        Start of event loop. Get input. Do stuff.
        Send to GPU, present frame (input is 4ms out of date)
12ms    Done.
        Start of event loop. Get input. Do stuff.
        Send to GPU, present frame (input is 4ms out of date)
16ms ----------------------

What I'd like bevy to do with vsync on:

0ms -----------------------
        Sleep until we have just enough time to render a frame, based on how long it took previously.
12ms    Start of event loop. Get input. Do stuff.
        Send to GPU, present frame (input is 4ms out of date)
16ms ----------------------

We could prototype this by adding a system at the end of the event loop that sleeps for a while after the frame has been sent to the GPU. I'm not intimately familiar with how all that works, but I can try something out.

[bjorn3]

I believe mailbox vsync should behave similar to disabled vsync in terms of latency.

What I'd like bevy to do with vsync on:

This is called frame pacing, right? You will have to be careful to ensure that variation in render time doesn't cause frames to be submitted too late. Predicting the right time to sleep can be difficult.

[aevyrie]

I believe mailbox vsync should behave similar to disabled vsync in terms of latency.

That's what I would expect, but not what I experience in bevy apps.

Predicting the right time to sleep can be difficult.

Definitely. My use case for this would be in applications though, I care more about reducing input latency without just letting the app run at 300fps, draining battery. In fact, for the application use case you could safely add a 2x safety factor to predicted frame render time, and still get a huge improvement in latency without risking dropped frames, because applications like this take very little time to render - on the order of 1-3ms.

[minecrawler]

Why sleep before the frame, if you can also sleep at the end?

0ms -----------------------
        Start of event loop. Get input. Do stuff.
12ms    Send to GPU, present frame (input is 4ms out of date)
        Sleep until the remaining frame-time is over.
16ms ----------------------

To be honest, this is what I'd expect a frame-limiting strategy to look like. Just wait at the end until the specified time-frame is over, so that there is no timing issue later on in case the renderer is slower than expected :)

[aevyrie]

That doesn't get around the timing problem, presenting it this way just makes it seem like the problem doesn't exist. That's why I presented it in reverse - it makes the timing problem more apparent.

You need to make sure the time between "send to GPU" is always <16ms to prevent frame drops. In the order you present, you would need to add a sleep between the time the frame is finished and sent to the GPU to act as your factor of safety for any frame time variability. The "gotcha" here is that it seems like you can just sleep until you've hit a total of 16ms, because your total frametime is always 16ms, but it's masking the fact that what you actually care about is the time between "send to GPU" being 16ms.

Anywho, the proof is in the pudding. We should make some prototypes to see if we can make something that works. 😄

[aevyrie]

I did some work on this.

1. It appears we are still using FIFO vsync:
   

   bevy/crates/bevy_render/src/view/window.rs

   Lines 139 to 140 in ffecb05

   	present_mode: if window.vsync {
   	wgpu::PresentMode::Fifo

   
   See revert default vsync mode to Fifo #1416

2. I made a frame limiter app that adds a frame limiting system in the renderer sub app, in the final cleanup stage. You can check it out here: https://github.com/aevyrie/bevy_latency

It works like this:

0ms -----------------------
        Start of event loop. Get input. Do stuff.
 4ms    Send to GPU, present frame (input is 4ms out of date)
New!    Run a stopwatch in a system in RenderStage::Cleanup to time how long a frame takes (not including sleep)
New!    Sleep the thread for the predicted amount of time, with some safety factor added to prevent frame drops, 
            in case it actually takes longer to render the frame than predicted.
16ms ----------------------

Here's a trace to better visualize, notice the large blue bar with the label "framerate limiter"

I don't have any empirical measurements, but so far it seems promising.

Mailbox vsync, framerate limiter enabled

bevy.2021-12-19.17-07-51_Trim_Trim.mp4

Mailbox vsync, framerate limiter disabled:

bevy.2021-12-19.17-08-14_Trim.mp4

With the framerate limiter, the 3d cursor feels perceptively less sluggish, but it's still not as good as with vsync off. I tested without vsync both with and without the framerate limiter, however I was still seeing some tearing with the framerate limited to ~60, though on the plus side I did see significantly less GPU/CPU usage.

[aevyrie]

I modified the prototype to add a second sleep system that caps the frame rate to exactly what you specify.

0ms -----------------------
            Estimate how long the next frame will take, minus a small margin to give us space if it takes longer
            Sleep for this duration. (Forward estimation)
8ms         Start of event loop. Get input. Do stuff.
11.8ms      See how close our estimate was to the requested frame time, sleep if required to get the frametime just right
12ms        Send to GPU, present frame (input is 4ms out of date)
16ms ----------------------

Red annotation: forward estimation. Blue annotation: precise frametime limiter accounting for error and margin in the last frame's forward estimation.

I'm seeing some really awesome results, the 3d cursor is basically glued to the mouse cursor:

bevy.2021-12-19.23-48-46_Trim.mp4

In addition, I can bring my safety margin pretty low without frame drops - on the order of 100μs. I've bumped it up to 500μm to reduce chances of frame drops, but at the cost of only 400μs more input lag.

Edit: this wasn't possible without spin_sleep to get precise sleep times. Thanks for the suggestion @cwfitzgerald!

[aevyrie]

Some other neat byproducts of this, it's now really easy to framelimit to an arbitrarily low FPS for power use or other reasons. However, because our input -> render latency is constant (3.5ms in my case), the game/app still feels really responsive at low framerates! Here's the demo locked to only 20fps, yet the 3d cursor still doesn't lag behind the OS cursor very much:

bevy.2021-12-20.01-42-14.mp4

It doesn't feel laggy or jello-y, because the motion-to-photon time is still low, instead it only feels choppy because it doesn't update very frequently.

This brings up an interesting idea for system scheduling too. If a game or application is sensitive to input responsiveness, but needs a large frametime budget, they could schedule everything compute intensive after the render stage, instead of between input and the render stage. This means those changes would take up to a full frametime to display, but you now have the ability to only put critical things (like transforming objects in the world based on user input) in the pre-render schedule.