-
-
Notifications
You must be signed in to change notification settings - Fork 304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improving input latency for the FIFO present mode #819
Draft
mitchmindtree
wants to merge
2
commits into
nannou-org:master
Choose a base branch
from
mitchmindtree:fifo-lower-latency
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After landing nannou-org#782 I noticed that the `App` proxy's `wakeup` method hasn't been functional since the last event loop refactor (when wgpu was introduced). I also noticed that nannou-org#782 makes the extra updates provided in the `Wait` loop mode redundant. Previously, we ensured at least ~3 updates would occur following the most recently received event. This was aimed at allowing GUI's to update for a couple of extra frames to resolve subtle 1-3 frame animations, but I'm unsure this is really the right approach. At the very least, the number of updates should be configurable as a part of the `LoopMode::Wait` variant, but that can wait for a future PR.
This begins work on improving the latency between user input and when the window's surface texture is presented to the display for wgpu's FIFO present mode. Note that you can already improve latency by providing a `SurfaceConfigurationBuilder` with `wgpu::PresentMode::Mailbox` specified which is available on *most* platforms. This PR however aims specifically at improving the `Fifo` present mode as it is the default present mode and only mode guaranteed by WGPU to work across *all* platforms. **The Problem** The primary issue is that under the `Fifo` present mode, requesting the surface texture to draw to can block for up to a frame (~16ms at 60fps) depending on how recently the last frame's surface texture was submitted. Once the surface texture is acquired, we then draw to it and must wait up to another frame before that texture is presented to the window on the user's display. This means we can frequently get up to 2 frames (~33ms at 60fps) worth of total latency between user input and visualisation of that input. This is particularly noticable when controlling a camera with a mouse, or interacting with any kind of GUI. **Potential Solutions** Ideally, it would be nice if `wgpu::Surface::get_current_texture` provided a non-blocking alternative, or some way to query whether or not requesting a new surface texture would block. I imagine this would allow us to continue to collect new user input in the mean-time and perform an `update` right at the moment the new texture is acquired. This would allow us to cut down on up to a frame worth of input latency. In lieu of this we can get close to the same behaviour by attempting to predict the moment we expect a new surface texture to be available and avoid calling `get_current_texture` until we believe it would no longer block. We can do so by keeping track of the moment at which we acquire each texture, and then returning `ControlFlow::WaitUntil(next_frame)` where `next_frame` is the moment we acquired the last texture *plus* the duration of a frame interval. This is the approach currently taken in this PR, and does result in a noticable improvement in input latency. **Caveats & TODO** The duration of a frame differs between displays based on their refresh rates. E.g. the minimum frame interval of a 60hz display is ~16ms, for a 144hz display it is ~7ms. Currently, this PR just assumes 16ms as a proof-of-concept, however ideally we'd retrieve the actual refresh rate from somehwere. `winit` does provide a way to query the *supported* video modes (and in turn, refresh rates) of the monitor upon which a window is currently placed, however it does not appear possible to retrieve the *active* refresh rate. One option might be to simply use the interval duration of the highest rate to avoid missing any frames, at the risk of retaining some latency in the case that a lower refresh rate is active. This solution gets much fuzzier when we start to think about apps with multiple windows across multiple displays with varying refresh rates (not uncommon for installations where you hav a GUI/control window and one or more visualisation windows across different displays/projectors). Resolving nannou-org#817 would help to make this a little clearer.
Woyten
reviewed
Dec 14, 2021
loop_start: Instant, | ||
last_update: Instant, | ||
total_updates: u64, | ||
events_since_wakeup: usize, | ||
last_surface_texture_acquired: Option<std::time::Instant>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think using the instant
crate would be better in order to be compatible with WASM.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on #818.
This begins work on improving the latency between user input and when
the window's surface texture is presented to the display for wgpu's FIFO
present mode.
Note that you can already improve latency by providing a
SurfaceConfigurationBuilder
withwgpu::PresentMode::Mailbox
specified which is available on most platforms. This PR however aims
specifically at improving the
Fifo
present mode as it is the defaultpresent mode and only mode guaranteed by WGPU to work across all
platforms.
The Problem
The primary issue is that under the
Fifo
present mode, requesting thesurface texture to draw to can block for up to a frame (~16ms at 60fps)
depending on how recently the last frame's surface texture was
submitted. Once the surface texture is acquired, we then draw to it and
must wait up to another frame before that texture is presented to the
window on the user's display. This means we can frequently get up to 2
frames (~33ms at 60fps) worth of total latency between user input and
visualisation of that input. This is particularly noticable when
controlling a camera with a mouse, or interacting with any kind of GUI.
Potential Solutions
Ideally, it would be nice if
wgpu::Surface::get_current_texture
provided a non-blocking alternative, or some way to query whether or not
requesting a new surface texture would block. I imagine this would allow
us to continue to collect new user input in the mean-time and perform an
update
right at the moment the new texture is acquired. This would allowus to cut down on up to a frame worth of input latency.
In lieu of this we can get close to the same behaviour by attempting to
predict the moment we expect a new surface texture to be available and
avoid calling
get_current_texture
until we believe it would no longerblock. We can do so by keeping track of the moment at which we acquire
each texture, and then returning
ControlFlow::WaitUntil(next_frame)
where
next_frame
is the moment we acquired the last texture plus theduration of a frame interval. This is the approach currently taken in
this PR, and does result in a noticable improvement in input latency.
Caveats & TODO
The duration of a frame differs between displays based on their refresh
rates. E.g. the minimum frame interval of a 60hz display is ~16ms, for a
144hz display it is ~7ms. Currently, this PR just assumes 16ms as a
proof-of-concept, however ideally we'd retrieve the actual refresh rate
from somehwere.
winit
does provide a way to query the supportedvideo modes (and in turn, refresh rates) of the monitor upon which a
window is currently placed, however it does not appear possible to
retrieve the active refresh rate. One option might be to simply use
the interval duration of the highest rate to avoid missing any frames,
at the risk of retaining some latency in the case that a lower refresh
rate is active.
This solution gets much fuzzier when we start to think about apps with
multiple windows across multiple displays with varying refresh rates
(not uncommon for installations where you hav a GUI/control window and
one or more visualisation windows across different displays/projectors).
Resolving #817 would help to make this a little clearer.