Change ffmpeg argument order so that it correctly cuts at keyframes

[avh4]

When I run ffmpeg with the arguments in this order -i ... -ss ... -t ..., it produces a video file that has no video for a few initial seconds. (as far as I can understand this is because it doesn't seek to a keyframe?)

If I change the order to -ss ... -i ... -t ..., then the resulting cut videos play properly.

My `ffmpeg -version`

ffmpeg version 3.2 Copyright (c) 2000-2016 the FFmpeg developers
built with Apple LLVM version 8.0.0 (clang-800.0.38)
configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.1.2/include/openjpeg-2.1 --enable-nonfree --enable-vda
libavutil      55. 34.100 / 55. 34.100
libavcodec     57. 64.100 / 57. 64.100
libavformat    57. 56.100 / 57. 56.100
libavdevice    57.  1.100 / 57.  1.100
libavfilter     6. 65.100 /  6. 65.100
libavresample   3.  1.  0 /  3.  1.  0
libswscale      4.  2.100 /  4.  2.100
libswresample   2.  3.100 /  2.  3.100
libpostproc    54.  1.100 / 54.  1.100

[codyolsen]

I've had this problem too. Thanks for the work on the fix @avh4!

[avh4]

Hmm... well, I'm not sure this is actually better yet... After this change, the resulting mp4 files play correctly on Mac, but if I process them further with ffmpeg, there are still problems with the initial few seconds. I will try to experiment with this further.

[avh4]

So I'm having the best results atm with

      '-noaccurate_seek',
      '-ss', cutFrom,
      '-i', filePath, '-y', '-vcodec', 'copy', '-acodec', 'copy',
      '-to', cutTo,
      '-f', format,
      '-avoid_negative_ts', 'make_zero',

I'll try to set up a script to compare the effect of some of the different changes here to see what's actually best.

[mifi]

Good job! I've got some test videos/audios that I used to test with:
https://github.com/mifi/lossless-cut-fixtures
Note that many of them actually don't work in the original version either.

[avh4]

So it seems that the -noaccurate_seek is what does it.. it will cut at the nearest keyframe, but that means the cut time may be different from what you choose (in my case, by as much as 8 seconds). Because the keyframe may be before the seek time you specified, the -avoid_negative_ts make_zero adjusts all the timestamps in the output so they aren't negative (which for me caused problems later in my processing pipeline).

So I guess I'm not sure lossless-cut would want to add the -noaccurate_seek flag. It makes the times different from what you select, but then it also makes the start of the output video usable. (Another option is that ffprobe -select_streams v -show_frames <videofile> can be used to find out where the keyframes are, so possibly the UI could only let you choose cut points from the set of keyframe locations?)

I'm still investigating ways to generate a keyframe at the desired cutpoint in order to produce an output that both is cut accurately and starts with a keyframe.

[mifi]

Will the audio be shifted to be in sync with the video when using -noaccurate_seek and -avoid_negative_ts ?

I don't think it's a problem that the actual cut time is different from what you specified. Because as you say without it the part before the keyframe is useless anyways, so it has the same effect.

I actually thought about recreating the lost part after the first keyframe too, by maybe doing a re-encoding of the part between the cutpoint and the first keyframe, and then losslessly merge this part with the cut part. But then it would have to be encoded using the same parameters as the original video.

Edit: Regarding ffprobe -select_streams v -show_frames <videofile>, I was really close to implementing this (i have it in a local branch). However i found that this operation is very slow, especially for big files, so it defeats the purpose of the program being a quick way to process videos.

https://trac.ffmpeg.org/wiki/Seeking

[avh4]

So the problem with using -noaccurate_seek is that it will cut the video at the nearest keyframe, but the audio will be cut at the specified time.. so the resulting file will have an initial segment of audio with no video, and then the video will start at the keyframe. That seems bad too :( It seems like the only way around that is to figure out where the keyframe is and then tell ffmpeg to cut at that point.... Yeah, ffprobe is quite slow... it seems to take ~1 minute per GB for me. Maybe there are options for it, or another tool that can scan for keyframe times faster?

[mifi]

So is there any difference between using -noaccurate_seek and not using it? from what i can understand from your last comment, you will get the first part without video in both cases?

There has to be a faster way of detecting keyframes. If you look propietary software, they are able to generate thumbnails (presumably from keyframes) almost instantly throughout the timeline.

[jpaulos]

From the user experience side of things, you might want to play with how the Avidemux approach feels. They "encourage" the user to split on keyframes by having the video slider automatically jump to keyframes. They also make the more obvious forward/reverse buttons be keyframe jumps. However, there are also frame-by-frame advance arrows. No idea what's going on at the backend, I'm just a "user."

[mifi]

Thanks for the input. I agree that we should snap to or show keyframes somehow, but I don't know how to quickly and easily get keyframe positions from the video. Using ffprobe is very slow as previously discussed.

[Lazza]

I'm wondering if Smart Rendering could be implemented to solve this problem in the best possible way. In the past I tried to script ffmpeg and do some computations with Python scripts but it didn't work perfectly. I know for sure that Smart Cutter does this, however it's an illegal ripoff of Mencoder, FFmpeg and Xvid without any source released, so it cannot be studied.

[mifi]

How did you do it in your python script?

I was thinking about doing something like this:
http://apple.stackexchange.com/a/176474/187402
See Lossless (but way more complicated!)

However the same problem arises, which is that ffprobe is very slow.

[Lazza]

How did you do it in your python script?

The idea is the classic Smart Rendering approach, e.g.:

A---[-B-----C--]--D

Assume A, B, C and D are I-frames:

• re-encode [-B using similar video quality (you can get the quality level with mediainfo, IIRC)
• cut B-----C using -codec copy
• re-encode C--] as above
• use the concat filter to merge the pieces

It was almost working, then I realized not every I-frame is a keyframe (some GOPs contain back references even across I-frames) and you can only cut on "true" keyframes. But you cannot determine what the real keyframes are with ffprobe or other methods. So my code would work randomly 1 out of maybe 3 times, not very useful. Finally, I gave up.

ffprobe is very slow

What I did was to cut 2 minutes of video around my point of interest (e.g. [) with -codec copy, then use ffprobe on that piece and do the math to translate I-frame positions back to the original video. That was not very difficult, but the problem with real keyframes remains.

[mifi]

Great explanation, thanks. That's the same way I thought of doing it.

I tried to use the read_intervals options for ffprobe, which acheives the same by seeking before printing, but it's still very slow. Just printing frame info for 10 seconds of video takes like 20 seconds on my MBP, which is way too slow.

ffprobe -print_format json -select_streams v -show_frames -read_intervals '20%+10' test.mov

[Lazza]

Have you tried with a video file which is actually 10 seconds long?

[mifi]

Yes, I tried with a DJI footage. it took 15 sec.

[Lazza]

I see. Well, personally I would happily wait 1 or 2 more minutes when cutting a hour long video, if I knew I would get smart rendering... but of course other people might have different opinions on this. 😄

Also, maybe you could run ffprobe in the background when a user is dragging the marker around a certain neighborhood? That would enhance the perceived speed I guess.

[mifi]

Maybe. But it complicates things a bit. Best would be to figure out how to somehow read all keyframes superfast like Avidemux apparently does.

I found a way to speed up ffprobe a bit btw. By providing -skip_frame nokey
However it is still not very fast. maybe a 4x speed increase

[WAZAAAAA0]

If this is of any help, FFmpegYAG also known as FFmpeg Hi GUI has a way to distinguish I/P/B frames seemingly instantaneously just like Audacity, except... well, it uses FFmpeg.

https://sourceforge.net/projects/ffmpegyag/

[mifi]

I've tried to use the ffmpeg command line to cut by first using ffprobe with -read_intervals to find the nearest I-frame and then pass its time value to ffmpeg -i file.mp4 -ss timevalue ...
However, when cutting on the I-frame timestamp, ffmpeg skips the desired keyframe and leaves seconds of empty video until the next keyframe. Even when going one frame back in time (the frame before I-frame), it still skips the desired keyframe. What I found kind of works is cutting 3 frames behind the I-frame. It cuts perfectly on some videos. However not on all videos. I have a local branch which implements this in LosslessCut: ffprobe-find-nearest-keyframe
I have tried to understand how ffmpeg seeking -ss really works, but not able to grasp it. And I have no idea why 3 frames behind I-frame is the magic number.

I also discovered that -noaccurate_seek only applies when encoding. It does nothing when doing -c copy.

One option might be to use the "segment" format in ffmpeg, as it seems to cut by keyframe. but it complicates things.

[mifi]

I tried to use the ffmpeg segment output format with the default segment time, but it also produces segments with the wrong length (empty video in the end of each segment), similar to cutting with ffmpeg -ss time -i file ....

[Lazza]

Your findings about ffmpeg match my experiments I did in the past. That's also because not every I-frame is a keyframe, unfortunately... 😭

[eexpress]

I think we need add two tool icons, which jump to last/next keyframe. like in avidemux.
wait for issues26 and issues13 solve.
I like this KISS software.

[mifi]

Yes that should be fairly trivial to implement but not very useful until we figure out how to actually cut on keyframes

[modest]

FYI, I believe there is a slightly hacky way to get even closer to what you're trying to accomplish.

While it's true that you must trim at GOP boundaries, you don't necessarily need to display all of those frames. In the mp4 headers that you write, you can "mute" frames at the beginning and end by setting their duration to 0 in the stts (Sample to Time) table.

These 0-duration frames will be available for reference by other frames, but will not play. (Make sure to drop audio frames appropriately, too.)

The "right" way to do this is with mp4/mov Edit List metadata (elst box), which were specifically designed for lossless video trimming. Unfortunately they aren't well-supported in .mp4, so compatibility varies…

[mifi]

🆕🆕In the newest version 1.13.0 I've added a button to the leftmost of the right button row, called nc (normal cut). If you toggle this to kc (keyframe cut) before doing the cut, then it will instead cut at the closest keyframe, -ss before -i, as discussed and suggested by @avh4 , and the output will not contain an empty portion at the beginning of the file. Because I also use -avoid_negative_ts make_zero in that case, audio should be fine

👆 Would be nice if people can try this feature and see if this causes any trouble when processing files further!

I will probably make this the default in the future, as it seems to be working nicely. Unless someone reports some trouble with it.

[TechManiacHD]

So far from my observations it seems that new feature works perfectly fine. Audio is synced with video 👍 I didn't check on too many files as most of my stuff is in HEVC. Soon I'll test more.

Thank You :)

[eexpress]

The new functions is very good. But those small buttons are too small and a little hard to understand. Would you like to change them into menu mode, tick menu? At least the menu text is more clearer than hover one.

[mifi]

@eexpress I agree, the buttons are a bit tricky. A menu would be really nice, but I have to figure out how to easily dynamically update menu state. Thinking about something like https://github.com/SamyPesse/react-electron-menu but it seems a bit unmaintained.

[eexpress]

@eexpress I agree, the buttons are a bit tricky. A menu would be really nice, but I have to figure out how to easily dynamically update menu state. Thinking about something like https://github.com/SamyPesse/react-electron-menu but it seems a bit unmaintained.

For a consistent user interface, maybe put those options here (just your help interface) is more easy (maybe temporarily), and it also easy to dynamically update those text.

[AgostinoSturaro]

Works for me. Full HD 60FPS .mov files from an Olympus Stylus.
Audio seems in sync.

I agree the interface is a bit confusing, and I would easily forget to set the kc option every time.
Specifically, using labeled radio buttons would be better than toggles.
Toggles are usually associated with features being ON/OFF.

Still, thanks for implementing this.

[TechManiacHD]

• Toggles ON/OFF would look clearer :)
• What about "auto-save settings"?

[GeoDecouvreTout]

To summarize, the nc (normal cut) & kc (keyframe cut) cutting modes correspond respectively to the following examples:

ffmpeg -i input.mp4 -ss 00:27 -t 00:15 -acodec copy -vcodec copy -y output_nc.mp4

ffmpeg -ss 00:27 -i input.mp4 -t 00:15 -avoid_negative_ts make_zero -acodec copy -vcodec copy -y output_kc.mp4

The next link suggests combining seeking (-ss before AND after -i, with a -ss position before AND a -ss duration after -i as explained):
https://trac.ffmpeg.org/wiki/Seeking
i.e.
ffmpeg -ss 00:20 -i input.mp4 -ss 00:07 -t 00:15 -avoid_negative_ts make_zero -acodec copy -vcodec copy -y output_new.mp4

Here, I use also -avoid_negative_ts make_zero to obtain the same nc mode output file but "without" the empty portion problem. In fact, in this case, VLC does not play the empty portion.

To be checked and tested on your application cases, with your video players, to see if combining seeking really helps...

[ccoenen]

Regarding "finding keyframes", there's a way other than ffprobe. I'm not sure if it is much faster, but it also allows selectively searching (as opposed to processing the whole file in one attempt):

ffmpeg -i input.mp4 -vf select='between(t,10,20)*eq(pict_type,I)',showinfo -f null -

select documentation, including more examples
showinfo documentation

• between(t,10,20) means "time between 10 and 20 seconds" (which is one filter)
• * (combined with next filter)
• eq(pict_type,I) pict_type must be equal to "I", meaning it is a keyframe.

This yields output like this (which apparently can't be further customized), but at least it includes the timestamp. The frame number can't be trusted, because they get renumbered after the filter.

[Parsed_showinfo_1 @ 0000023956b4e6c0] n:   0 pts: 360360 pts_time:12.012  pos:  3715891 fmt:yuv420p sar:1/1 s:1280x720 i:P iskey:1 type:I checksum:9A735DDB plane_checksum:[39437D88 09F5C8BA 18FA178A] mean:[122 125 129] stdev:[51.6 9.6 11.4]
[Parsed_showinfo_1 @ 0000023956b4e6c0] n:   1 pts: 450450 pts_time:15.015  pos:  4820727 fmt:yuv420p sar:1/1 s:1280x720 i:P iskey:1 type:I checksum:C01D3823 plane_checksum:[C62B5865 659B5FFB CCA57FB4] mean:[117 125 128] stdev:[45.3 10.1 10.7]
[Parsed_showinfo_1 @ 0000023956b4e6c0] n:   2 pts: 540540 pts_time:18.018  pos:  5904619 fmt:yuv420p sar:1/1 s:1280x720 i:P iskey:1 type:I checksum:FAEBD0BA plane_checksum:[FDD1FBD2 04440D04 E500C7D5] mean:[108 126 129] stdev:[45.1 10.0 9.0]

[mifi]

@GeoDecouvreTout whats the Point of -ss both before and after?

[mifi]

@ccoenen yea we could run this +- a few seconds around current time when moving the cursor and update a separate list of known keyframe times

[GeoDecouvreTout]

From my understanding:

With kc (-ss before -i), the video starts at the closest keyframe. In this case, there is no empty portion at the beginning of the file, but the actual starting point can be distant from the desired cursor position.

With nc (-ss after -i), the actual starting point is closer to the desired cursor position, but with an empty portion at the beginning. Depending on the video players, their release and the file format, this empty portion may be played or not.

With -ss before and after -i , the actual starting point is identical to the nc cut (close to the desired cursor position) but here the empty portion seems to be never played (to be confirmed). To be honest, I don't know why. I'm not a video expert. It's a simple observation on several video clips.
The drawback of this approach is that the choice of the -ss position (before -i) and the -ss duration (after -i) is arbitrary (several possibilities). Once the -ss values have been determined to start as close as possible to the desired cursor position, the -t duration must be adjusted to end the video at your convenience. So implementing this cutting mode would probably lead to manage three cursor positions instead of two.

[sanmathigb]

I stumbled upon this extremely relevant thread while trying to resolve very similar problems discussed above, the difference being that the seeking functionality is being implemented within my c++ code and not via the ffmpeg command line. So I am working with av_seek_frame and the AV_FLAG_BACKWARD flags. I am running into a/v sync issues and inaccurate seeking. Any pitfalls to avoid here or any relevant prior results/ findings shared will be really helpful.
Thanks!

[friendlygiraffe]

This is really helpful, using -ss ... -i ... -t ... gets them much closer to the keyframe.

Is there a way to repair movie files that have been cut using -i ... -ss ... -t ... ? I have lot of files cut using this method, and wondered if there was an easy way to fix them

[GeoDecouvreTout]

If you mean repairing movie files cut with the nc mode in order to get the result you would have obtained with the kc mode, I’m afraid it’s not possible because nc output files are usually smaller than kc output files (not enough data to “repair” them).

However, you can try to apply the kc mode with -ss 00:00 to your nc files in order to delete the empty portion. It sometimes works (possible workaround):

ffmpeg -ss 00:00 -i input_nc.mp4 -avoid_negative_ts make_zero -acodec copy -vcodec copy -sn -map_metadata -1 -y output_kc.mp4

Besides, this second ffmpeg pass could be implemented as a nkc mode (for example) in LosslessCut.

I have also noticed that the nc mode works generally better when audio is deleted (da). This empty portion problem is clearly related to audio as already mentioned.

[friendlygiraffe]

If you mean repairing movie files cut with the nc mode in order to get the result you would have obtained with the kc mode, I’m afraid it’s not possible because nc output files are usually smaller than kc output files (not enough data to “repair” them).

However, you can try to apply the kc mode with -ss 00:00 to your nc files in order to delete the empty portion. It sometimes works (possible workaround):

ffmpeg -ss 00:00 -i input_nc.mp4 -avoid_negative_ts make_zero -acodec copy -vcodec copy -sn -map_metadata -1 -y output_kc.mp4

Besides, this second ffmpeg pass could be implemented as a nkc mode (for example) in LosslessCut.

I have also noticed that the nc mode works generally better when audio is deleted (da). This empty portion problem is clearly related to audio as already mentioned.

This works! Thanks

[mifi]

closing in favor of #1216