😍 Implement "smart cut"

[mifi]

We cannot cut between keyframes, it's a technical limitation of modern video codecs. So ffmpeg will choose the nearest keyframe, which will not be the exact time you requested.

Might be able to use avcut for h264 although it is experimental. See #372 (comment)

See also discussion in #13

Update: Have made an initial implementation of this. This experimental feature will re-encode the part of the video from the cutpoint until the next keyframe in order to attempt to make a 100% accurate cut while losslessly copying the rest of the segment.

VideoRedo

VideoRedo is a now-defunct software that had feature overlap with LosslessCut.

• https://www.videohelp.com/software/VideoReDo
• https://forum.videohelp.com/threads/407142-VideoReDo-owner-passed-away-product-ceased-seeking-alternatives#post2668738

Smart Media Cutter

Recently built in order to replace VideoRedo

They open sourced their smart cutting algorithm, could take inspiration from it: https://github.com/skeskinen/smartcut

Remaining issues

Inaccurate seeking

Seeking is not always accurate, causing glitches and/or jumps in the stitch points. This is often because I haven't found a way to consistently seek to and cut on an exact frame. Sometimes it could be caused by the "audio samples" problem below. If you want to experiment, you can try to remove the audio track and cut only video to see if it makes a difference.

• How to seek to and cut from a frame in ffmpeg? #1216

Audio samples problem

Cut time accuracy/glitches when audio samples are large? (try without audio and see if it makes a difference). A workaround seems to be to cut audio/video separately and then merge them, with the tradeoff of potential audio/video de-sync:

• Smart Cut - option to cut audio separately in whole, then combine with video result (Request) #1834
• Audio becomes "stretched" around cut from Smart cut only with H264+AAC MP4 files. #2050

subtitles

• Using Smart Cut makes Subtitles unsynchronized  #2049

Other improvements

• Smart Cut Custom Settings #1824
• If there is no keyframe in a particular segment, re-encode the whole segment instead of erroring with Cannot find any keyframe after the desired start cut point, see #Smart cut failure: "Unable to find keyframe" #1190 (comment)

Done

• Implement a smart cut that re-encodes (lossy) the part between the cutpoint and next keyframe. (done)
• Video quality gets degraded when "smart cutting" (experimental) #1606
• Investigate using aomav1 instead libsvtav1 instead of aomav1 (ffmpeg default) (codec needs to compiled for mac in ffmpeg-build-script) Change smart cut AV1 encoder to libsvtav1 #1825
• custom bit rate SmartCut extreme quality degradation for encoded part due to incorrect detected bitrate from stream #1997

Limitations

• Only works on some files:
  • I've had success with some h264 files, and only a few h265 files seem to work.
  • Other codecs may or may not work.
• Only supports cutting a single video stream (track). All other video streams have to be disabled. We could still probably do multi-passes and then recreate a file with all the streams.
  • Thumbnail stream will be dropped
• We scan up to 60 seconds to find a keyframe.
• It tries to auto-detect bitrate from input stream, however if it fails, then it will calculate bitrate from the whole file's size/duration (may be wrong)
• Tries to copy timebase from source to destination

[ilaw138]

Hi guys,

Interesting that it is a limitation of the codecs. Reason I say this is that I can do a frame specific "lossless" cut using Apple's (very old) Quicktime Player 7 in Pro mode. For the cut to take at the specific frame it has to export to a new file (rather than edit the existing file). But you can do this just passing through the source data with out any form of recoding (both video and audio tracks).

Not sure exactly how it does it, but seems that som,ehting could be possible. Save time isn't too bad either as it just seems to shuttle the data into a new file.

Happy to send you a before and after file of a minute or so duration showing what it can do, if you'd like that.

This is about the only thing that I would need to replace the ancient Quicktime Player 7 with lossless-cut as my mainstay tool. Other enhancemnets wouldbe nice, but this one is pretty essential for frame level edits.

Hope this helps.

Cheers,

Ian

[mifi]

Have you tried quicktime pro with a large file? Like 1gb, does it still take only a couple of seconds?
If it's quick then I'm thinking maybe they do a smart cut, re-encoding only the inter-keyframe portion

[DGrv]

This would be awesome really :)

[CoeNT]

I more or less grew up with SvenOver's DVBcut which did implement "keyhole surgery" editing. He never updated it for HD (H.264 and etc) and, in the end, I shifted to VideoReDo which handles top-and-tailing and advert removal very efficiently and reliably using "smart cut" techniques and FFmpeg as the internal engine. Alas, it's Windows-only, not open-source and non-free (but relatively cheap) but it's a good benchmark to follow. So ... I'm absolutely enthralled if you're planning to add the facility to your LosslessCut ... all power to your elbow!

In the meantime though, I wonder if you might consider the following: There has been some discussion here already on "cut at the natural frame-break before or after the desired cut-point " (so apologies if the matter has already been settled). How about a pair of extra buttons adjacent to the < and > buttons that skip to the natural break respectively before or after? That way the user can choose (to taste) whether to exclude the last (intrusive) advert frame or include the first (treasured) movie frame. At the moment, it's not obvious which sacrifice one will make :-).

I'm enjoying getting to know your creation - it's a valuable addition to the linux toolkit. Thank you!

David

I'd like to point out that if you use -ss to cut at a keyframe after the input, the resulting video doesn't seem to play the first segment correctly. Setting it to a tenth of a second before the keyframe does work, but it does go to show that using -ss before the input is superior.

It's a quirk to keep in mind when implementing smart cut.

[pkamb]

So ffmpeg will choose the nearest keyframe

Does this ever lose frames? If there are key frames at frames 10 and 15, and your trim starts at frame 13, does it take the *nearest keyframe?

Do you lose the first 2 frames of video? Or does it look for the first keyframe before/after your in/out points?

[danielmee]

Seeing as @ilaw138 didn't reply, I can. I've been using QTP7 and MPEG Streamclip for the past 10 years to manage the media files of a weekly show at work. It's possible to do a perfect cut on any size file, here are the steps:

• Trim to the exact position you want using the in/out markers in QTP7
• Save it as a separate file (you can use QTP7's reference file option though so it will save a new file however fast your computer can write to disk and the size of the file)
• Open the file in MPEG Streamclip and select File > Save As
• In the save dialogue box change the dropdown to be mp4 and Save.
  Done, no reencoding, exact to the frame and plays as expected. Feel free to ask me to provide you anything you need that might help you to solve this issue.

[mifi]

Hi. Thanks for the info. I'm not really sure how QuickTime Pro does this, and I don't really understand why you need to use Streamclip in addition to quicktime pro?

[ilaw138]

Apologies. Missed your e-mail about large files. In short, no I haven’t tried to edit out segments frame exact on a large file of 1GB. QTP7 didn’t really like large files in any event - at least in my experience - so never tried anything too challenging for the software. Have found if I save file from QTP7 (as opposed to export) it didn’t cut the frames exactly - so assume using key-frame - but export does. Sorry I can’t be of more help. Regards, Ian

…

On 14 Feb 2020, at 12:55, Daniel Mee ***@***.***> wrote: Seeing as @ilaw138 <https://github.com/ilaw138> didn't reply, I can. I've been using QTP7 and MPEG Streamclip for the past 10 years to manage the media files of a weekly show at work. It's possible to do a perfect cut on any size file, here are the steps: Trim to the exact position you want using the in/out markers in QTP7 Save it as a separate file (you can use QTP7's reference file option though so it will save a new file however fast your computer can write to disk and the size of the file) Open the file in MPEG Streamclip and select File > Save As In the save dialogue box change the dropdown to be mp4 and Save. Done, no reencoding, exact to the frame and plays as expected. Feel free to ask me to provide you anything you need that might help you to solve this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#126?email_source=notifications&email_token=ADIUWJYFHDTB5JX5QPTQ2MDRC2IFLA5CNFSM4GWMKI2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELY54EY#issuecomment-586276371>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADIUWJ2G4QRDHZRPB6U6CJLRC2IFLANCNFSM4GWMKI2A>.

[danielmee]

... and I don't really understand why you need to use Streamclip in addition to quicktime pro?

Quicktime doesn't let you save a clipped file as a .mp4, only a .mov, Streamclip allowed me to get it back to .mp4 without recompressing. The only other way to get it out of QTP7 as a .mp4 was to export it but then it would compress it to those export settings.

[danielmee]

QTP7 didn’t really like large files in any event - at least in my experience
Have found if I save file from QTP7 (as opposed to export) it didn’t cut the frames exactly - so assume using key-frame - but export does.

Not meaning to contradict you Ian but my experience working with QTP7 for 10 years pretty much every day was totally different. It regularly handled the 5, 10 or even 50G files I threw at it and its accuracy was spot on to the frame each time. If it wasn't I wouldn't have been able to use it as the transitions between segments in the program had to be edited out to the frame and QTP7 worked flawlessly each time. I also used the info panels to find out the duration to the millisecond (see the screenshot of a video's details, note the Current Time)

If it's helpful I can do a screencast of myself using it and how accurate it was... if it's going to help.