FFMPEG concat issue - Manim produces empty media and silently ignores warning

[jeertmans]

Description of bug / unexpected behavior

For some configurations, Manim will produces empty media (i.e., videos) as partial movie files.

Because Manim currently calls FFMPEG in subprocesses, we are not aware of all the warnings it might raise, because the FFMPEG command line utility does a lot of "cleaning" to avoid potential issues.

In link to #3501, I also decided to use PyAV for my Manim Slides project. Recently, a user reported an interesting issue that only occurs on very rare cases, see jeertmans/manim-slides#390. This issue occurs when we try to concatenate media files that include an empty video media.

If we try to run the ffmpeg command (that is run by Manim), but in the command-line, we observe a few interesting warnings (see logs below). An important message in the output is the deprecation warning, which means that Manim may break in the future if we do not do anything.

In an effort to debug my implementation of concat (and others), I decided to move the development of those ffmpeg/av filters to a separate project: avfilters. The idea is to create a higher-level interface to libav for Manim Slides (and Manim?), while testing the audio-video functions on a large test-suite inside the avfilters project. After many tests, it really looks like the best would be to avoid generating empty media.

Expected behavior

Not generating empty media files, because implementing a fix for handling possibly empty media files in concat seems much more complex than just not generating empty media (which apparently seems to be the only issue).

How to reproduce the issue

Check the MWE from jeertmans/manim-slides#390 (comment).

Using Manim Slides is not important, as Manim will still generate 5 media files, and one of them will be empty. You can download the files for reproduction: https://github.com/jeertmans/avfilters/tree/b172c4f728a299a35c6b1488f85a809e52918056/tests/issues/manim-slides-390.

Logs

Terminal output

Running the following command:

ffmpeg -f concat -safe 0 -i media/videos/mwe/1080p60/partial_movie_files/MWE/partial_movie_file_list.txt -c copy output.mp4 -y

yields:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5589f8155e80] Auto-inserting h264_mp4toannexb bitstream filter
Input #0, concat, from 'media/videos/mwe/1080p60/partial_movie_files/MWE/partial_movie_file_list.txt':
  Duration: N/A, start: 0.000000, bitrate: 57 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 57 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Output #0, mp4, to 'output.mp4':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, q=2-31, 57 kb/s, 60 fps, 60 tbr, 15360 tbn, 15360 tbc
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=    1 fps=0.0 q=-1.0 size=       0kB time=-00:00:00.03 bitrate=N/A speed=N/A    
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5589f815cb80] Auto-inserting h264_mp4toannexb bitstream filter
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5589f81ffbc0] Auto-inserting h264_mp4toannexb bitstream filter
    Last message repeated 1 times
DTS -141670994486043792, next:2966667 st:0 invalid dropping
PTS -141670994486043280, next:2966667 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486043536, next:2983334 st:0 invalid dropping
PTS -141670994486042256, next:2983334 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486043280, next:3000001 st:0 invalid dropping
PTS -141670994486042768, next:3000001 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486043024, next:3016668 st:0 invalid dropping
PTS -141670994486043024, next:3016668 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486042768, next:3033335 st:0 invalid dropping
PTS -141670994486042512, next:3033335 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486042512, next:3050002 st:0 invalid dropping
PTS -141670994486041232, next:3050002 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486042256, next:3066669 st:0 invalid dropping
PTS -141670994486041744, next:3066669 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486042000, next:3083336 st:0 invalid dropping
PTS -141670994486042000, next:3083336 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486041744, next:3100003 st:0 invalid dropping
PTS -141670994486041488, next:3100003 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486041488, next:3116670 st:0 invalid dropping
PTS -141670994486040208, next:3116670 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486041232, next:3133337 st:0 invalid dropping
PTS -141670994486040720, next:3133337 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486040976, next:3150004 st:0 invalid dropping
PTS -141670994486040976, next:3150004 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486040720, next:3166671 st:0 invalid dropping
PTS -141670994486040464, next:3166671 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486040464, next:3183338 st:0 invalid dropping
PTS -141670994486039184, next:3183338 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486040208, next:3200005 st:0 invalid dropping
PTS -141670994486039696, next:3200005 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486039952, next:3216672 st:0 invalid dropping
PTS -141670994486039952, next:3216672 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486039696, next:3233339 st:0 invalid dropping
PTS -141670994486039440, next:3233339 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486039440, next:3250006 st:0 invalid dropping
PTS -141670994486038160, next:3250006 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486039184, next:3266673 st:0 invalid dropping
PTS -141670994486038672, next:3266673 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486038928, next:3283340 st:0 invalid dropping
PTS -141670994486038928, next:3283340 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486038672, next:3300007 st:0 invalid dropping
PTS -141670994486038416, next:3300007 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486038416, next:3316674 st:0 invalid dropping
PTS -141670994486037136, next:3316674 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486038160, next:3333341 st:0 invalid dropping
PTS -141670994486037648, next:3333341 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486037904, next:3350008 st:0 invalid dropping
PTS -141670994486037904, next:3350008 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486037648, next:3366675 st:0 invalid dropping
PTS -141670994486037392, next:3366675 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486037392, next:3383342 st:0 invalid dropping
PTS -141670994486036112, next:3383342 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486037136, next:3400009 st:0 invalid dropping
PTS -141670994486036624, next:3400009 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486036880, next:3416676 st:0 invalid dropping
PTS -141670994486036880, next:3416676 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486036624, next:3433343 st:0 invalid dropping
PTS -141670994486036368, next:3433343 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486036368, next:3450010 st:0 invalid dropping
PTS -141670994486035088, next:3450010 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486036112, next:3466677 st:0 invalid dropping
PTS -141670994486035600, next:3466677 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486035856, next:3483344 st:0 invalid dropping
PTS -141670994486035856, next:3483344 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486035600, next:3500011 st:0 invalid dropping
PTS -141670994486035344, next:3500011 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486035344, next:3516678 st:0 invalid dropping
PTS -141670994486034064, next:3516678 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486035088, next:3533345 st:0 invalid dropping
PTS -141670994486034576, next:3533345 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486034832, next:3550012 st:0 invalid dropping
PTS -141670994486034832, next:3550012 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486034576, next:3566679 st:0 invalid dropping
PTS -141670994486034320, next:3566679 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486034320, next:3583346 st:0 invalid dropping
PTS -141670994486033040, next:3583346 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486034064, next:3600013 st:0 invalid dropping
PTS -141670994486033552, next:3600013 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486033808, next:3616680 st:0 invalid dropping
PTS -141670994486033808, next:3616680 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486033552, next:3633347 st:0 invalid dropping
PTS -141670994486033296, next:3633347 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486033296, next:3650014 st:0 invalid dropping
PTS -141670994486032016, next:3650014 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486033040, next:3666681 st:0 invalid dropping
PTS -141670994486032528, next:3666681 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486032784, next:3683348 st:0 invalid dropping
PTS -141670994486032784, next:3683348 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486032528, next:3700015 st:0 invalid dropping
PTS -141670994486032272, next:3700015 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486032272, next:3716682 st:0 invalid dropping
PTS -141670994486030992, next:3716682 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486032016, next:3733349 st:0 invalid dropping
PTS -141670994486031504, next:3733349 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486031760, next:3750016 st:0 invalid dropping
PTS -141670994486031760, next:3750016 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486031504, next:3766683 st:0 invalid dropping
PTS -141670994486031248, next:3766683 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486031248, next:3783350 st:0 invalid dropping
PTS -141670994486029968, next:3783350 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486030992, next:3800017 st:0 invalid dropping
PTS -141670994486030480, next:3800017 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486030736, next:3816684 st:0 invalid dropping
PTS -141670994486030736, next:3816684 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486030480, next:3833351 st:0 invalid dropping
PTS -141670994486030224, next:3833351 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486030224, next:3850018 st:0 invalid dropping
PTS -141670994486028944, next:3850018 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486029968, next:3866685 st:0 invalid dropping
PTS -141670994486029456, next:3866685 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486029712, next:3883352 st:0 invalid dropping
PTS -141670994486029712, next:3883352 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486029456, next:3900019 st:0 invalid dropping
PTS -141670994486029200, next:3900019 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486029200, next:3916686 st:0 invalid dropping
PTS -141670994486028176, next:3916686 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486028944, next:3933353 st:0 invalid dropping
PTS -141670994486028688, next:3933353 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
DTS -141670994486028688, next:3950020 st:0 invalid dropping
PTS -141670994486028432, next:3950020 invalid dropping st:0
[mp4 @ 0x5589f81e8dc0] pts has no value
frame=  240 fps=0.0 q=-1.0 Lsize=      46kB time=00:00:03.95 bitrate=  94.9kbits/s speed= 271x    
video:43kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 7.421508%

FFMPEG

Output of ffmpeg -version:

4.4.2-0ubuntu0.22.04.1

[behackl]

Since #3491 has been merged, the MWE posted over at jeertmans/manim-slides#390 raises

ValueError: AnimationGroup(Group) has a run_time of 0 seconds, this cannot be rendered correctly. Please set the run_time to be positive.

I consider this to be the correct behavior: animations with a run_time of 0 indicate an issue with the scene as they practically do not really make sense. I'm not sure whether it is still possible to generate these files (even more exotic approaches like Scene.wait_until(lambda: True) produce a one-frame long, non-empty partial movie file).

I think we might be able to simply close this issue.

[lucmos]

Thanks for the reply!

I sometimes add self.wait() at the end of the animation to ensure it completes correctly (sometimes there is some leftover during e.g. uncreate), and this ensure it is not displayed (important for some slides)

I believe that sometimes this is not needed, and in the previous version would produce an empty animation. Is my understanding correct/the extra wait handled without raising an error? 😁

[behackl]

A plain self.wait() always produces a 1-second animation (and playing them at the end usually does make sense; due to the way the rendering loop works internally, the last frame rendered from an animation displays the penultimate state of the involved mobjects) -- as far as I can tell the cause of the 0-second animation is the attempt of playing an empty AnimationGroup.

[behackl]

I tried some more, didn't manage to produce an empty partial movie file with the check from #3491 in place. Let's close this for now, and come back to it in case we get reports of corrupted pipes or corresponding av-errors again.

[jeertmans]

Sure, thanks @behackl!