Post-processing videos (still frame removal and speedup)

[sgbaird]

This can be standalone from #212, but is of course related. The idea would be to remove segments of the video where there is little to no change (which could be difficult when there are blinking lights for example) and run a speedup of e.g., 16x, to later be uploaded to a separate playlist (we can worry about automatic video uploads later, though not sure if that requires just an API key or needs token.pickle).

https://claude.ai/share/217878d2-d194-4441-8c91-9af937ba787b

[sgbaird]

Per our conversation, I think the decision was to include something in the title (maybe original video id? Or could be timestamp) that we can keep the same between the original and the processed videos, and add some kind of note in the title like: [processed, 16x]. @Neil-YL

This can be used with the YouTube API, grabbing lists of the video names from from original and processed. From the YouTube API quota perspective, if multiple pages are returned, each page will count towards the quota.

Per @Jonathan-Woo 's suggestion, hosting this on a paid-tier HuggingFace space with a the lowest tier GPU (could also try the free, preemptable A100) and the minimum sleep time of 5 minutes of inactivity seems pretty reasonable and low-cost.

@Jonathan-Woo which algorithms/codebases were you looking at? I had found some in the transcript above (a few links were dead of course), might be worth a quick look.

[sgbaird]

@Jonathan-Woo here's the example video you can use: https://www.youtube.com/live/Tbru5BiokmU?si=Sbmfu5dggSYLpnsp

[Jonathan-Woo]

Yep, I looked at the transcript, thanks for that. I'm looking into motion and auto-editor right now. Both can provide timestamps. I think I'll test both and examine their performance. Leaning more towards motion as it seems to be more popular.

[sgbaird]

Related: https://docs.frigate.video/ (I guess @yakavetsiv is incorporating or at least has plans to). Thanks @kelvinchow23 for the ping about it

EDIT: using with esp32 cameras, PoE, and local network

Ilya running everything on a local server, Zima Cube (frigate, MQTT broker, Prefect local)

[Jonathan-Woo]

I've been using auto-editor so far and the results are pretty good on the video linked above. The following command cut these sections out:

https-www-youtube-com-live-Tbru5BiokmU-si-WtsDaMc0DSHxFbKH_ALTERED.mp4

The tricky part is to tune:

1. margin: amount of stale video added to edited sections to smooth out edit
2. threshold: % of "motion" required to not be stale

There is also support for yt-dlp directly so we can pass the stream link directly. Also, there is support for hardware encoding.

auto-editor https://www.youtube.com/live/Tbru5BiokmU\?si\=WtsDaMc0DSHxFbKH --edit motion:threshold=0.2 --video-speed 99999 --silent-speed 1 -c:v h264_videotoolbox --download-format bv --margin 10sec

One issue is that, since we don't have a wall clock, it's difficult to tell when video is being sped up. So, I'm working on adding an indicator for when video is being sped up. Then, it will be easier to tune the settings.

[sgbaird]

Really cool to see this! Had thought about adding an overlay directly via the pi zero 2w but not sure if it will handle that extra processing well, and might complicate other things (for one, we lose the raw footage). Ultimately leaned away from it since YouTube already has timestamps.

White text with slightly transparent black background has often worked well from a contrast and aesthetics point of view.

Any luck with adding the indicator?

Also would be great if you could share the sections that were kept, if not too much work.

[sgbaird]

@Jonathan-Woo bump

[Jonathan-Woo]

Really cool to see this! Had thought about adding an overlay directly via the pi zero 2w but not sure if it will handle that extra processing well, and might complicate other things (for one, we lose the raw footage). Ultimately leaned away from it since YouTube already has timestamps.

White text with slightly transparent black background has often worked well from a contrast and aesthetics point of view.

Any luck with adding the indicator?

Also would be great if you could share the sections that were kept, if not too much work.

Still working on adding the indicator. I'm using ffmpeg to add an overlay similar to https://video.stackexchange.com/questions/12105/add-an-image-overlay-in-front-of-video-using-ffmpeg.

Overall, this my plan:

1. auto-editor to get v1 file to get timestamps for when to add overlay.
2. ffmpeg to add overlay
3. auto-editor to accelerate stale sections

This is the kept video:

video_short_ALTERED.mov

[sgbaird]

Awesome, thanks! It seems like it did a pretty good job with separating the stale vs. non-stale

auto-editor to accelerate stale sections

Quick aside: Originally, I was imagining deleting the stale sections entirely and speeding up the non-stale sections

[Jonathan-Woo]

Sure that's totally possible. We would just have to update the --silent-speed argument.

[Jonathan-Woo]

Overlay has been added.

overlayed.mp4

I'm concerned about removing stale sections entirely because I'm concerned that this editor may have false positives.

I will share the final sped up version soon. In the meantime we should think about what to remove/speed up.

[sgbaird]

Nice! Good point about false positives. I'm worried a bit about devices that have a lot of stale time (e.g., 8 hrs of stale video @ 16x = 30 min), which might get cumbersome for promo vids and robotic training. Though, it could cause problems if people are doing post-mortem analysis (i.e., watching the sped-up video) and miss something that would otherwise catch their eye.

I could be missing something else too. I should probably document and consolidate the various intended uses somewhere (these have made it on a whiteboard occasionally).

Aside: is the intention to put the speedup factor in the overlay? (e.g., 16x).

[Jonathan-Woo]

This is the edited version with the overlay indicating 16x. It still flickers a bit so the sensitivity governed by the motion will likely need to be tuned more.

Other parts of the pipeline left is the communication with youtube in terms of downloading, uploading, organizing into playlists, etc. Let's sync tomorrow?

edited.mp4