Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to capture replays #579

Merged
merged 38 commits into from
Oct 8, 2024
Merged

Add option to capture replays #579

merged 38 commits into from
Oct 8, 2024

Conversation

kmagiera
Copy link
Member

@kmagiera kmagiera commented Oct 1, 2024

This PR adds replays support.

Replays allows the user to capture a recent moment into a video and preview it instantly within the IDE panel.

The current way this feature has been designed is as follows:

  • in device setting you can enable/disable replays functionality. Disabling hides the replays button from the UI
  • we're adding new "replay" button that's in the main IDE panel and can be use to capture the replay and moves the user into the preview mode
  • in preview mode we're displaying a full screen overlay over the IDE panel with video player
  • the overlay has a seekbar, arrows to move forward/backward by one frame, save button, and the video length selector
  • simulator-server uses a fixed size memory buffer for replays, depending on the frame rate (it is variable) it may store a different lengths of the video depending on the content. When replay is captured we show the past 5 seconds by default and allow user to increase that length using length selector as long as the captured video is of a sufficient duration.

Here's what we are adding as part of this PR:

  1. New UI element to PreviewView that displays the preview button
  2. New UI screen that's responsible for rendering replay preview with all the controls
  3. We're extending the webview<>extension API with methods for capturing video replays and for saving video files
  4. We're extensing device settings with the new replay setting. Then we use this setting to start/stop recording depending on device session state. We want the recording to start after every device reload, such that we're not recording reload screens etc.

Test plan

  1. Enable replays in device options -> see it changes the appearance of replay button in top-right
  2. Tap on replay after a while, the video should appear in an overlay
  3. For longer videos, you should be able to change between lengths
  4. Updating length once video finished playback should make it play from a certain moment
  5. Updating length to shorter when player is at the start of playback, should move to the start of new length
  6. Save button should store the full length video. When one is stored with the same name, you'll get a confirmation dialog

Copy link

vercel bot commented Oct 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
radon-ide ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 8, 2024 10:20am
react-native-ide ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 8, 2024 10:20am

Copy link
Collaborator

@filip131311 filip131311 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing I came across a problem with menage devices menu that is probably caused by the overlay:

Screen.Recording.2024-10-07.at.17.47.00.mov

@filip131311
Copy link
Collaborator

review notes:

  • Replays do not work when option was selected while iPhone is actively simulated
  • the name of saved file should not include whitespaces
  • I believe the setting enabling replay should be located in top settings menu as it is not a device setting, but ide setting

className="switch-root small-switch"
id="enable-replays"
onCheckedChange={(checked) =>
project.updateDeviceSettings({ ...deviceSettings, replaysEnabled: checked })
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I belive this option should be part of workspace settings (we have a separate interface for that called WorkspaceConfig as it is not a device setting also it would be nice if this option synchronized between computers for users that are logged in to microsoft account

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about it, but feel like it fits the device setting pane better for now.

The workspace settings doesn't have any configuration options except from the IDE panel location for now. Adding it there would make it an isolated option. For me it still feels like its device related because we're recording the device screen in the end. The only thing is that it affects the panel UI as it adds a new button when replays are enabled.

I'd keep it as is for now, and we can consider moving it in the future or once we have more similar options added that'd make it less of a precedence to have a switch button included in the workspace settings dropdown.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not agree, for the following reasons:

  • WorkspaceConfig currently includes showDeviceFrame setting that is affecting user ui in a similar way as replay button does, (it is not included in the settings dropdown but it is accessible through vscode settings panel)
  • it does not change anything on the device were as all other device settings do, I see your point that it records device but the recording does not happen on the device itself which means that it won't affect how device works during testing. As a user I would see the device setting menu as a menu i use often while testing my application and all other options as dose i set up only when something goes wrong.
  • this is a setting that is rarely used (probably only once per user) as a User does not have much reason to use it except their preference which I expect does not change often
  • workspace settings persist across devices for logged in user which is a useful functionality that we get for free
  • I'm not even sure if this setting needs to exist in the UI because of how rarely it should be used and it's location in the vscode settings would make it easy to remove it from UI settings if they get to cluttered

@kmagiera
Copy link
Member Author

kmagiera commented Oct 8, 2024

While testing I came across a problem with menage devices menu that is probably caused by the overlay:
Screen.Recording.2024-10-07.at.17.47.00.mov

Thanks for testing. This seemed to be some transient issue that got resolved after I merged main. Will look into other comments

@kmagiera
Copy link
Member Author

kmagiera commented Oct 8, 2024

Seems like @p-malecki resolved other minor comments.

The only part I don't understand is this comment:

Replays do not work when option was selected while iPhone is actively simulated

Can you provide more details on what exacly is not working here?

@kmagiera kmagiera merged commit d622e5a into main Oct 8, 2024
3 checks passed
@kmagiera kmagiera deleted the kmagiera/replays branch October 8, 2024 12:56
balins added a commit that referenced this pull request Oct 16, 2024
This PR fixes a bug introduced when adding video replays in #579 that
was making the timer freeze and sometimes show negative values during
rewind.

The issue was caused by updating only the video player's current time
and not propagating the change to the element rendering the overlay.
We've seemed to rely on `handleTimeUpdate` callback registered on
`"timeupdate"` event, but this event is not emitted when updating the
current time of the video programmatically.
Sometimes, the timer indicated negative values during rewind, and this
was also caused by setting the current time on video without updating
the overlay's `time` accordingly, which in turn led to `time = 0`
(default on mount) and `startTime = [video duration - 5]`, causing the
timer to show negative values when `video duration > 5` (we calculate
the time displayed on timer as `time - startTime`).

The proposed changes include passing a callback that allows the rewind
function not only for updating the video's current time, but also the
current time of the overlay component.
Another change is that we won't show `• REPLAY 0:00` until video
metadata is loaded to avoid flickering from `0:00` to `video duration`
when starting the rewind.


https://github.com/user-attachments/assets/fec826d1-b677-40fb-84cb-a13a792fe0ac


https://github.com/user-attachments/assets/b5f07a42-647d-4c34-8fc0-126df618a637

### How Has This Been Tested: 

Tested that with introduced changes, the timer behaves as expected both
in rewind and normal playback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants