fix: stream input coordinates use true frame dimensions - #1614
Open
kevingatera wants to merge 3 commits into
Open
fix: stream input coordinates use true frame dimensions#1614kevingatera wants to merge 3 commits into
kevingatera wants to merge 3 commits into
Conversation
The stream server substituted its configured viewport (default 1280x720) for the screencast frame's deviceWidth/deviceHeight, but the actual content area differs whenever window chrome, scrollbars, emulation, or a manual resize shrink or grow it (observed: 1280x633 for a 1280x720 window). Clients map input coordinates into this DIP space, so the fabricated values made every click land off target, scaled by the ratio between the phantom and true dimensions. Forward the real CDP metadata and only fall back to the configured viewport when CDP omits the values; the e2e test now skips not-yet-settled frames before asserting.
The dashboard scaled clicks and wheel events by the configured viewport from the status message, which can differ from the real content area, so pointer input landed off target (23px low on a 1280x720 window whose content area is 1280x633). Track the per-frame deviceWidth/deviceHeight from the (now real) CDP metadata and map input into that DIP space, falling back to the configured viewport before the first frame arrives. Document the coordinate space in the README and streaming docs.
Contributor
|
@kevingatera is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
Conflict in cli/src/native/stream/cdp_loop.rs: upstream vercel-labs#1594/vercel-labs#1626/vercel-labs#1627 restructured frame delivery (seq + latest-frame-wins watch channel, configurable screencast size, ms timestamps). Kept upstream's new delivery path and reapplied the coordinate fix on top: frame metadata forwards the real CDP deviceWidth/deviceHeight, falling back to the configured viewport when CDP omits them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In streaming mode, mouse clicks in the dashboard (or any WebSocket stream client) land off target whenever the browser's real content area differs from the configured viewport (#1613). A 1280x720 window produces a 1280x633 content area, so the dashboard stretched every Y coordinate by 720/633 and clicks landed proportionally too low.
Fix
cdp_loop.rsforwards the real CDPPage.screencastFramemetadatadeviceWidth/deviceHeightinstead of substituting the configured viewport, falling back only when CDP omits the valuesInput.dispatchMouseEventexpects; it falls back to the configured viewport before the first frame arrivesVerification
frame metadatareports 1280x633 after the fix, and a client mapping by metadata receivesCLICK 320,158, exactly on targetcargo testpasses,e2e_stream_frame_metadata_respects_custom_viewportpasses,cargo fmt --checkcleanFixes #1613.