Allow receiving an external frame index

[scotts]

🚀 The feature

We have had several request for the ability for VideoDecoder to accept an external frame index upon startup. That is, in exact mode, we build up a frame index which tells us the mapping between frame indices and pts. If users could provide that index to us, we could be in exact mode but not have to scan the entire file at startup. (We would, however, be the the mercy of the accuracy of the external index.)

Motivation, pitch

The main motivation is performance. There is potentially less need for this feature now that we have approximate mode (#427), which also avoids scanning the entire file. But the combination of receiving an external index and exact mode could perhaps yield the best of both worlds: exact seeks and fast startup.

The reason this comes up is that in many environments, videos are pre-processed before being used for training and inference. Creating and storing the external index for a particular video would be relatively low-cost, and it would be paid once.

[NicolasHug]

From a UX perspective, but also in terms of implementation, we may want to think of the user-provided index as a third seek_mode? The reason I say this is because when the index is provided by the user, we end up in a hybrid mode that doesn't really fit neither approximate nor exact. It's more of a combination of both:

• approximate: we don't scan, we don't rely on an index
• exact: we scan and rely on the index we built ourselves
• user-provided index: we don't scan (like in approx), but still use an index (like in exact).

[scotts]

That's a good point. I was thinking of the two concepts as being orthogonal, but you're right, they're related. That is, it doesn't make much sense to provide an external index if you're also using approximate mode - why provide us with something that you then ask us to not use? Ideally, our API would prevent such logical inconsistencies.

[Dan-Flores]

Additionally, we should consider which format(s) we want to support for an external frame index. One easy way to get a frame index is ffprobe's -show_frames command, which supports several formats including CSV and JSON.

ffprobe -i input.mp4 -select_streams v:0 -show_frames -print_format json

Ideally, I think we should support the most common format, or multiple common formats.

[scotts]

@Dan-Flores, good question! I actually see this as two questions:

1. What is the schema we should accept? That is, what data do we need, what are the types of that data, and how is it structured?
2. Given a schema, what format do we accept? This is the question of, say, CSV or JSON. The same schema can exist in multiple formats.

I do, however, have a single answer to both questions: I think we should do the schema returned by ffprobe, and we should use JSON. My reasoning:

1. I think ffprobe is how most people will generate such an index. Because folks are using TorchCodec, we know they have FFmpeg installed. The scenario that we're targeting here is when there is a large corpus of centrally stored videos to be used for training. It's relatively cheap and easy to also store a frame index with such videos to speed up the decoding at training time. Given that we assume FFmpeg will already be installed, it seems like calling ffprobe will be the easiest way for such folks to create such an index.
2. I assume JSON is the more popular format? When given a choice, I tend to prefer it because I like having explicit field names. I admit this point is more personal preference, and less a reasoned technical choice. :)

I also think it's fine to support just one format at first, and wait for folks to request additional formats. But that means we'll want to design the API in such a way that we can accept multiple formats.

[Dan-Flores]

I agree on both points @scotts! Given these two constraints, the ffprobe command should look something like this:

ffprobe -i $VIDEO_PATH -show_frames -print_format json >> $INDEX_FILE

There are two ways to build the schema from this JSON:

• The python VideoDecoder constructor could accept the JSON filepath then open and parse the needed information.
• Alternatively, we could provide a Python function to parse the JSON, then expect the output of that function in the VideoDecoder constructor.
  • Both options can also be implemented in C++, but this would add new dependencies since C++ lacks built-in JSON support.
  • Since the first option

The schema we use should be minimal, and contain the information we store during a scan in SeekMode::exact:

• key_frames: List of PTS of key frames
• all_frames: List of PTS of all frames
  • This tells us beginStreamPtsFromContent, numFramesFromContent
• duration: The duration of all frames
  • Using the duration of the last frame, we can get endStreamPtsFromContent.

This data can be converted to Tensors to continue to enable torch.compile, then get passed to the core ops and to the SingleStreamDecoder constructor:

 explicit SingleStreamDecoder(
      const std::string& videoFilePath,
      SeekMode seekMode = SeekMode::exact,
      std::optional<std::tuple<torch::Tensor, torch::Tensor, torch::Tensor>>
          externalFrameIndex = std::nullopt);

Then, initializeDecoder() will call a function to store the metadata.

For all SingleStreamDecoder functions, I believe we can reuse the behavior from SeekMode::exact, since we have effectively performed a scan, but please double check me on this

[NicolasHug]

I'd suggest to go ahead with the first option of accepting a file path as input. It may or may not be what we will land on for the final design, but that's simple enough to implement for a first iteration, and should unblock further progress.

For the schema, this is all private anyway so it's not critical, but instead of accepting key_frames as a list of pts:

key_frames: List of PTS of key frames

I think the key_frames list should be a list of booleans, of the same length as the all_frames list. Each boolean defines whether that frame is a keyframe or not. That's closer to the output of ffprobe, so it will require less preprocessing of the input:

~/dev/torchcodec (add-windows-cmake*) » ffprobe -v error -hide_banner -select_streams v:0 -show_frames test/resources/nasa_13013.mp4 | grep key
key_frame=1
key_frame=0
key_frame=0
...

For all SingleStreamDecoder functions, I believe we can reuse the behavior from SeekMode::exact, since we have effectively performed a scan, but please double check me on this

Correct, when the user passes their own index, the decoder should assume it to be the ground truth index and behave as if a scan was made.