Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving input latency for the FIFO present mode #819

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

mitchmindtree
Copy link
Member

Based on #818.

This begins work on improving the latency between user input and when
the window's surface texture is presented to the display for wgpu's FIFO
present mode.

Note that you can already improve latency by providing a
SurfaceConfigurationBuilder with wgpu::PresentMode::Mailbox
specified which is available on most platforms. This PR however aims
specifically at improving the Fifo present mode as it is the default
present mode and only mode guaranteed by WGPU to work across all
platforms.

The Problem

The primary issue is that under the Fifo present mode, requesting the
surface texture to draw to can block for up to a frame (~16ms at 60fps)
depending on how recently the last frame's surface texture was
submitted. Once the surface texture is acquired, we then draw to it and
must wait up to another frame before that texture is presented to the
window on the user's display. This means we can frequently get up to 2
frames (~33ms at 60fps) worth of total latency between user input and
visualisation of that input. This is particularly noticable when
controlling a camera with a mouse, or interacting with any kind of GUI.

Potential Solutions

Ideally, it would be nice if wgpu::Surface::get_current_texture
provided a non-blocking alternative, or some way to query whether or not
requesting a new surface texture would block. I imagine this would allow
us to continue to collect new user input in the mean-time and perform an
update right at the moment the new texture is acquired. This would allow
us to cut down on up to a frame worth of input latency.

In lieu of this we can get close to the same behaviour by attempting to
predict the moment we expect a new surface texture to be available and
avoid calling get_current_texture until we believe it would no longer
block. We can do so by keeping track of the moment at which we acquire
each texture, and then returning ControlFlow::WaitUntil(next_frame)
where next_frame is the moment we acquired the last texture plus the
duration of a frame interval. This is the approach currently taken in
this PR, and does result in a noticable improvement in input latency.

Caveats & TODO

The duration of a frame differs between displays based on their refresh
rates. E.g. the minimum frame interval of a 60hz display is ~16ms, for a
144hz display it is ~7ms. Currently, this PR just assumes 16ms as a
proof-of-concept, however ideally we'd retrieve the actual refresh rate
from somehwere. winit does provide a way to query the supported
video modes (and in turn, refresh rates) of the monitor upon which a
window is currently placed, however it does not appear possible to
retrieve the active refresh rate. One option might be to simply use
the interval duration of the highest rate to avoid missing any frames,
at the risk of retaining some latency in the case that a lower refresh
rate is active.

This solution gets much fuzzier when we start to think about apps with
multiple windows across multiple displays with varying refresh rates
(not uncommon for installations where you hav a GUI/control window and
one or more visualisation windows across different displays/projectors).
Resolving #817 would help to make this a little clearer.

After landing nannou-org#782 I noticed that the `App` proxy's `wakeup` method
hasn't been functional since the last event loop refactor (when wgpu was
introduced).

I also noticed that nannou-org#782 makes the extra updates provided in the `Wait`
loop mode redundant. Previously, we ensured at least ~3 updates would
occur following the most recently received event. This was aimed at
allowing GUI's to update for a couple of extra frames to resolve subtle
1-3 frame animations, but I'm unsure this is really the right approach.
At the very least, the number of updates should be configurable as a
part of the `LoopMode::Wait` variant, but that can wait for a future PR.
This begins work on improving the latency between user input and when
the window's surface texture is presented to the display for wgpu's FIFO
present mode.

Note that you can already improve latency by providing a
`SurfaceConfigurationBuilder` with `wgpu::PresentMode::Mailbox`
specified which is available on *most* platforms. This PR however aims
specifically at improving the `Fifo` present mode as it is the default
present mode and only mode guaranteed by WGPU to work across *all*
platforms.

**The Problem**

The primary issue is that under the `Fifo` present mode, requesting the
surface texture to draw to can block for up to a frame (~16ms at 60fps)
depending on how recently the last frame's surface texture was
submitted. Once the surface texture is acquired, we then draw to it and
must wait up to another frame before that texture is presented to the
window on the user's display. This means we can frequently get up to 2
frames (~33ms at 60fps) worth of total latency between user input and
visualisation of that input. This is particularly noticable when
controlling a camera with a mouse, or interacting with any kind of GUI.

**Potential Solutions**

Ideally, it would be nice if `wgpu::Surface::get_current_texture`
provided a non-blocking alternative, or some way to query whether or not
requesting a new surface texture would block. I imagine this would allow
us to continue to collect new user input in the mean-time and perform an
`update` right at the moment the new texture is acquired. This would allow
us to cut down on up to a frame worth of input latency.

In lieu of this we can get close to the same behaviour by attempting to
predict the moment we expect a new surface texture to be available and
avoid calling `get_current_texture` until we believe it would no longer
block. We can do so by keeping track of the moment at which we acquire
each texture, and then returning `ControlFlow::WaitUntil(next_frame)`
where `next_frame` is the moment we acquired the last texture *plus* the
duration of a frame interval. This is the approach currently taken in
this PR, and does result in a noticable improvement in input latency.

**Caveats & TODO**

The duration of a frame differs between displays based on their refresh
rates. E.g. the minimum frame interval of a 60hz display is ~16ms, for a
144hz display it is ~7ms. Currently, this PR just assumes 16ms as a
proof-of-concept, however ideally we'd retrieve the actual refresh rate
from somehwere. `winit` does provide a way to query the *supported*
video modes (and in turn, refresh rates) of the monitor upon which a
window is currently placed, however it does not appear possible to
retrieve the *active* refresh rate. One option might be to simply use
the interval duration of the highest rate to avoid missing any frames,
at the risk of retaining some latency in the case that a lower refresh
rate is active.

This solution gets much fuzzier when we start to think about apps with
multiple windows across multiple displays with varying refresh rates
(not uncommon for installations where you hav a GUI/control window and
one or more visualisation windows across different displays/projectors).
Resolving nannou-org#817 would help to make this a little clearer.
loop_start: Instant,
last_update: Instant,
total_updates: u64,
events_since_wakeup: usize,
last_surface_texture_acquired: Option<std::time::Instant>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using the instant crate would be better in order to be compatible with WASM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants