Refactor visualize.py with NiceGUI

[abrichr]

https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/visualize.py should be rewritten, maybe with https://nicegui.io/ , maybe something else.

We would also like to display performance information (see #85)

[ahmetutku]

For anyone trying to install NiceGUI, the installation process requires Rust and Cargo to compile extensions. You can download it from the link: https://rustup.rs/ or just by running "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh" in your terminal.

[ahmetutku]

Here is a possible implementation I drew:

[ahmetutku]

"I think a nice addition would be to modify it so that:

1. each parent InputEvent is a video that plays through its children, and
2. it displays performance information (as per the comment on github)"
   What does plays "through its children" mean? Also, the second point, performance information, is not currently completed right (this is an idea for future implementation)?

[abrichr]

@ahmetutku ,

Thank you for putting this together! Please run python -m puterbot.visualize. An example of what this looks like is on the README. To the left of the screen are screenshots corresponding to parent events. To the right of the screen are InputEvents. The top level input event is the parent. You will notice this is a tree structure.

each parent InputEvent is a video that plays through its children, and

To clarify, instead of displaying a single screenshot, we want to play a video of all screenshots associated with the parent event's children.

performance information, is not currently completed right (this is an idea for future implementation)?

Correct. However, this is currently being implemented, and may be ready shortly, so please think about how you would incorporate this. For the time being, however, just focus on re-implementing the current visualization with the video.

Please let me know if anything is unclear!

[ahmetutku]

@abrichr Sounds good. As the videos are relatively short, should they repeat playing like a GIF? Also, another implementation would be to add the whole recording to the top to give an overview of everything that was recorded. What are your opinions on this?

[abrichr]

I think the right approach is to use HTML canvas with requestAnimationFrame, but I am open to arguments to the contrary!

add the whole recording to the top to give an overview of everything that was recorded.

Good idea!