[BUG] Non-deterministic error: non monotonically increasing dts to muxer

[lucmos]

Description

Thanks for the great work (again) 😄

I am encountering a strange issue: sometimes the rendering fails when using Slide. When using the normal manim Scene, everything works.
The complete error is the following:

...
Concatenating animation files to 'slides/files/MWE' and generating reversed animations:   0%|          | 0/1 [00:00<?, ?it/s][03/13/24 10:00:40] INFO     Auto-inserting h264_mp4toannexb         utils.py:17
                             bitstream filter                                   
                    INFO     Auto-inserting     libav.mov,mp4,m4a,3gp,3g2,mj2:17
                             h264_mp4toannexb                                   
                             bitstream filter                                   
                    INFO     Auto-inserting h264_mp4toannexb         utils.py:24
                             bitstream filter                                   
                    INFO     Auto-inserting     libav.mov,mp4,m4a,3gp,3g2,mj2:24
                             h264_mp4toannexb                                   
                             bitstream filter                                   
                    INFO     Auto-inserting h264_mp4toannexb         utils.py:24
                             bitstream filter                                   
                    INFO     Auto-inserting     libav.mov,mp4,m4a,3gp,3g2,mj2:24
                             h264_mp4toannexb                                   
                             bitstream filter                                   
                    INFO     Auto-inserting h264_mp4toannexb         utils.py:24
                             bitstream filter                                   
                    INFO     Auto-inserting     libav.mov,mp4,m4a,3gp,3g2,mj2:24
                             h264_mp4toannexb                                   
                             bitstream filter                                   
                    ERROR    Application provided invalid, non       utils.py:31
                             monotonically increasing dts to muxer              
                             in stream 0: 252000 >=                             
                             -830103483316671820                                
                    ERROR    Application provided invalid, non      libav.mp4:31
                             monotonically increasing dts to muxer              
                             in stream 0: 252000 >=                             
                             -830103483316671820                                
╭───────────────────── Traceback (most recent call last) ──────────────────────╮                                             
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim/cli/render/commands.py:115 in render                             │
│                                                                              │
│   112 │   │   │   try:                                                       │
│   113 │   │   │   │   with tempconfig({}):                                   │
│   114 │   │   │   │   │   scene = SceneClass()                               │
│ ❱ 115 │   │   │   │   │   scene.render()                                     │
│   116 │   │   │   except Exception:                                          │
│   117 │   │   │   │   error_console.print_exception()                        │
│   118 │   │   │   │   sys.exit(1)                                            │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/slide/manim.py:97 in render                               │
│                                                                              │
│    94 │   │                                                                  │
│    95 │   │   config["max_files_cached"] = max_files_cached                  │
│    96 │   │                                                                  │
│ ❱  97 │   │   self._save_slides()                                            │
│    98                                                                        │
│    99                                                                        │
│   100 class ThreeDSlide(Slide, ThreeDScene):  # type: ignore[misc]           │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/slide/base.py:493 in _save_slides                         │
│                                                                              │
│   490 │   │   │                                                              │
│   491 │   │   │   # We only concat animations if it was not present          │
│   492 │   │   │   if not use_cache or not dst_file.exists():                 │
│ ❱ 493 │   │   │   │   concatenate_video_files(slide_files, dst_file)         │
│   494 │   │   │                                                              │
│   495 │   │   │   # We only reverse video if it was not present              │
│   496 │   │   │   if not use_cache or not rev_file.exists():                 │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/utils.py:31 in concatenate_video_files                    │
│                                                                              │
│   28 │   │                                                                   │
│   29 │   │   # We need to assign the packet to the new stream.               │
│   30 │   │   packet.stream = output_stream                                   │
│ ❱ 31 │   │   output.mux(packet)                                              │
│   32 │                                                                       │
│   33 │   input_.close()                                                      │
│   34 │   output.close()                                                      │
│                                                                              │
│ in av.container.output.OutputContainer.mux:211                               │
│                                                                              │
│ in av.container.output.OutputContainer.mux_one:232                           │
│                                                                              │
│ in av.container.core.Container.err_check:285                                 │
│                                                                              │
│ in av.error.err_check:336                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: [Errno 22] Invalid argument: 
'slides/files/MWE/ed830346964b7f46169140a001904a4053baa918f96755b3e4beaa1e42df89
23.mp4'; last error log: [mp4] Application provided invalid, non monotonically 
increasing dts to muxer in stream 0: 252000 >= -830103483316671820
[126925] Execution returned code=1 in 7.106 seconds returned signal null 

---

I managed to produce the following MWE that (sometimes) reproduces the issue:

import typing as tp
from functools import partial

from manim import *
from manim_slides import Slide
from powermanim import AutoActivable,  VGroupActivable


class MWE(Slide):

    def construct(self) -> None:
        input_space_x = VDict({"label": Tex("A"), "space": Circle(), "embedding": Square()})
        input_space_y = VDict({"label": Tex("B"), "space": Circle(), "embedding": Square()})

        inputspaces_label = Tex("Input Spaces")

        shape_activable = partial(
            AutoActivable,
            scale_active=None,
            active_fill_opacity=None,
            inactive_fill_opacity=None,
            active_stroke_opacity=1.0,
            inactive_stroke_opacity=0.15,
        )
        label_activable = partial(
            AutoActivable,
            scale_active=None,
            active_fill_opacity=1.0,
            inactive_fill_opacity=0.15,
            active_stroke_opacity=1.0,
            inactive_stroke_opacity=0.15,
        )

        all_elements = VGroupActivable(
            label_activable(inputspaces_label, group=0),
            label_activable(input_space_y["label"], group=0),
            shape_activable(input_space_y["space"], group=0),
            shape_activable(input_space_x["embedding"], group=1),
            shape_activable(input_space_y["embedding"], group=1),
        ).move_to(ORIGIN)

        self.add(all_elements)
        self.wait()

        for _ in range(len(all_elements)):
            self.play(all_elements.also_next())
        self.wait()

It is using powermanim, that is a small library where I store re-usable manim components. It does not do anything particular.

In this example, it is useful to determine which animations to play in sequence -- grouped by the group specified. E.g., useful to go over a bullet list or explain a diagram step-by-step. I fear that it is this re-ordering of previously created animations that causes issues.

---

Do you have any ideas on where the problem may be?
It seems to be in the reverse-animation generation.

Version

manim-slides, version 5.1.3

Platform

OS: EndeavourOS rolling rolling
Kernel: x86_64 Linux 6.7.8-arch1-1
Uptime: 2d 1h 18m
Packages: 1364
Shell: zsh 5.9
Resolution: 3840x1200
DE: KDE 5.115.0 / Plasma 5.27.10
WM: KWin
GTK Theme: Breeze [GTK2/3]
Icon Theme: breeze
Disk: 121G / 140G (91%)
CPU: Intel Core i7-1065G7 @ 8x 3.9GHz [67.0°C]
GPU: Mesa Intel(R) Iris(R) Plus Graphics (ICL GT2)
RAM: 9006MiB / 15768MiB

Screenshots

Additional information

• It happened also other times, but I did not manage to produce a MWE. It is quite tricky because it is non-deterministic, I am not sure why.

• It seems to happen more often when animations are created, but then played in different order.

• In the past, I worked around the problem by adding AnimationGroups and reducing the number of .play if I remember correctly -- Having generalized some logic into a library it is no longer possible though

[jeertmans]

Hello! Thanks for opening this issue!

This issue is caused by the fact that I recently move from calling FFMPEG to using PyAV. While this is much better for performances and portability, this has the disadvantage that I do not understand everything I do with PyAV (documentation is quite poor) and I am afraid that I do not handle all the cases (like all possible video formats, etc).

I am not sure to have time for that before next week (I need to prepare a conference presentation for next Tuesday).

In the meantime, do you think you could produce a MWE that does not depend on your custom library? This is important so that I can later include this MWE in the tests suite.
Thanks!

[lucmos]

Thanks! 🚀

The following is a self-contained MWE. I also added some comments on which changes solve the error. I couldn't really figure out why, though.

import typing as T
from collections import defaultdict

from manim import *
from manim_slides import Slide


class VGroupActivable(VGroup):
    def __init__(
        self,
        *args: T.Sequence[T.Tuple[VMobject, int]],
    ) -> None:
        objects, groups = list(zip(*args))
        super().__init__(*objects)

        self.group2items: Dict[int, T.Set[VMobject]] = defaultdict(set)
        for obj, group in zip(objects, groups):
            self.group2items[group].add(obj)

        self.previously_active_idxs = []

    @property
    def ngroups(self) -> int:
        return len(self.group2items)

    def activate(self, indices: T.Union[int, T.Sequence[int]]) -> AnimationGroup:
        anims = []

        for to_activate in indices:
            # Somehow, commenting out this if (that only skips some anim...), solves the issue
            if to_activate in self.previously_active_idxs:
                continue
            anims.append(
                AnimationGroup(*(Transform(x, x.copy().set_opacity(1)) for x in self.group2items[to_activate]))
            )

        self.previously_active_idxs = indices

        return AnimationGroup(*anims)


class MWE(Slide):
    def construct(self) -> None:
        N = 5
        elements = (
            VGroupActivable(
                *((Dot(), i % 2) for i in range(N * N)),
            )
            .arrange_in_grid(N, N)
            .move_to(ORIGIN)
            .set_opacity(0.15)
        )

        self.add(elements)
        self.wait()

        # Setting this to range(elements.ngroups), that is correct in this case, solves the issue
        # Still with Scene instead of Slide everything works nevertheless
        for i in range(3):
            self.play(elements.activate(list(range(i + 1))))

        # Somehow, commenting out this wait solves the issue
        self.wait()

---

The following is the error when using Slide:

Concatenating animation files to 'slides/files/MWE' and generating reversed animations:   0%|          | 0/1 [00:00<?, ?it/s][03/13/24 12:26:10] ERROR    Application provided invalid, non       utils.py:31
                             monotonically increasing dts to muxer              
                             in stream 0: 252000 >=                             
                             -830103483316671820                                
╭───────────────────── Traceback (most recent call last) ──────────────────────╮                                             
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim/cli/render/commands.py:115 in render                             │
│                                                                              │
│   112 │   │   │   try:                                                       │
│   113 │   │   │   │   with tempconfig({}):                                   │
│   114 │   │   │   │   │   scene = SceneClass()                               │
│ ❱ 115 │   │   │   │   │   scene.render()                                     │
│   116 │   │   │   except Exception:                                          │
│   117 │   │   │   │   error_console.print_exception()                        │
│   118 │   │   │   │   sys.exit(1)                                            │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/slide/manim.py:97 in render                               │
│                                                                              │
│    94 │   │                                                                  │
│    95 │   │   config["max_files_cached"] = max_files_cached                  │
│    96 │   │                                                                  │
│ ❱  97 │   │   self._save_slides()                                            │
│    98                                                                        │
│    99                                                                        │
│   100 class ThreeDSlide(Slide, ThreeDScene):  # type: ignore[misc]           │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/slide/base.py:493 in _save_slides                         │
│                                                                              │
│   490 │   │   │                                                              │
│   491 │   │   │   # We only concat animations if it was not present          │
│   492 │   │   │   if not use_cache or not dst_file.exists():                 │
│ ❱ 493 │   │   │   │   concatenate_video_files(slide_files, dst_file)         │
│   494 │   │   │                                                              │
│   495 │   │   │   # We only reverse video if it was not present              │
│   496 │   │   │   if not use_cache or not rev_file.exists():                 │
│                                                                              │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-pac │
│ kages/manim_slides/utils.py:31 in concatenate_video_files                    │
│                                                                              │
│   28 │   │                                                                   │
│   29 │   │   # We need to assign the packet to the new stream.               │
│   30 │   │   packet.stream = output_stream                                   │
│ ❱ 31 │   │   output.mux(packet)                                              │
│   32 │                                                                       │
│   33 │   input_.close()                                                      │
│   34 │   output.close()                                                      │
│                                                                              │
│ in av.container.output.OutputContainer.mux:211                               │
│                                                                              │
│ in av.container.output.OutputContainer.mux_one:232                           │
│                                                                              │
│ in av.container.core.Container.err_check:285                                 │
│                                                                              │
│ in av.error.err_check:336                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: [Errno 22] Invalid argument: 
'slides/files/MWE/4d76edfd2f650dfaefd100d8c7d1f3f24eec24c3573a86b464248d03b36fbe
c6.mp4'; last error log: [mp4] Application provided invalid, non monotonically 
increasing dts to muxer in stream 0: 252000 >= -830103483316671820
[199460] Execution returned code=1 in 2.476 seconds returned signal null 

---

The following is the animation created successfully when using Scene:

MWE.mp4

[jeertmans]

Ok so just took a look for a few minutes, but av is quite hard to use so I will just develop the needed audio/video filters in a separate library (https://github.com/jeertmans/avfilters) so that I don't need to test that inside Manim Slides.

The idea will be to move concatenate and reverse utilities to this module, as well as other filters that I implemented for Manim's PR.

I'll ping back here once I have done some work on this :-)

[lucmos]

I am also hitting this error -- I think the root cause may be related.

The video is correctly produced, however manim-slides fails after all animations have been played.

                    INFO     Rendered DataExperiment4                                                   scene.py:241
                             Played 23 animations
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-packages/manim/cli/rend │
│ er/commands.py:115 in render                                                                     │
│                                                                                                  │
│   112 │   │   │   try:                                                                           │
│   113 │   │   │   │   with tempconfig({}):                                                       │
│   114 │   │   │   │   │   scene = SceneClass()                                                   │
│ ❱ 115 │   │   │   │   │   scene.render()                                                         │
│   116 │   │   │   except Exception:                                                              │
│   117 │   │   │   │   error_console.print_exception()                                            │
│   118 │   │   │   │   sys.exit(1)                                                                │
│                                                                                                  │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-packages/manim_slides/s │
│ lide/manim.py:97 in render                                                                       │
│                                                                                                  │
│    94 │   │                                                                                      │
│    95 │   │   config["max_files_cached"] = max_files_cached                                      │
│    96 │   │                                                                                      │
│ ❱  97 │   │   self._save_slides()                                                                │
│    98                                                                                            │
│    99                                                                                            │
│   100 class ThreeDSlide(Slide, ThreeDScene):  # type: ignore[misc]                               │
│                                                                                                  │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-packages/manim_slides/s │
│ lide/base.py:493 in _save_slides                                                                 │
│                                                                                                  │
│   490 │   │   │                                                                                  │
│   491 │   │   │   # We only concat animations if it was not present                              │
│   492 │   │   │   if not use_cache or not dst_file.exists():                                     │
│ ❱ 493 │   │   │   │   concatenate_video_files(slide_files, dst_file)                             │
│   494 │   │   │                                                                                  │
│   495 │   │   │   # We only reverse video if it was not present                                  │
│   496 │   │   │   if not use_cache or not rev_file.exists():                                     │
│                                                                                                  │
│ /home/luca/micromamba/envs/latentcommunication_anims/lib/python3.11/site-packages/manim_slides/u │
│ tils.py:18 in concatenate_video_files                                                            │
│                                                                                                  │
│   15 │   f.close()                                                                               │
│   16 │                                                                                           │
│   17 │   input_ = av.open(f.name, options={"safe": "0"}, format="concat")                        │
│ ❱ 18 │   input_stream = input_.streams.video[0]                                                  │
│   19 │   output = av.open(str(dest), mode="w")                                                   │
│   20 │   output_stream = output.add_stream(                                                      │
│   21 │   │   template=input_stream,                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: tuple index out of range

[jeertmans]

I think this might be related to ManimCommunity/manim#3648.
I have developed https://github.com/jeertmans/avfilters and will soon release a new version of Manim Slides that will use it and skip empty media files, at least until ManimCommunity/manim#3648 is fixed.

[jeertmans]

Hello @lucmos, v5.1.4 was just released, feel free to give it a try as it should solve your issue :-)

Please ping me if that is not the case so I can reopen this!

[lucmos]

Thank you so much for fixing this! 😄
Everything seems to be working with no hard failures now 🚀