Live stream decoding

[scotts]

🚀 The feature

Currently, both audio and video decoding depend on knowing both the total number of frames and the duration of the stream. As a consequence, we can't decode live streaming video or audio because we cannot know the total number of frames or the duration.

This feature would be to somehow enable decoding of live media. Because the APIs for VideoDecoder and AudioDecoder assume the ability to seek to arbitrary places in the stream, I suspect we might need a new top-level public API specifically for live streams.

Motivation, pitch

On the training side, I suspect this feature will not be used much. That is, training tends to happen on large corpuses of pre-existing media. I doubt there's much demand to train on a live stream. But doing inference on a live stream seems like it would be a common use-case, and we can't currently support it.

[FredrikNoren]

We're interested in this. We're doing research on agents in video games, and for that we'd like to have a stream of frames from the game which we send to our policy.

@scotts As a workaround meanwhile; do you think it would work if we set the duration to some arbitrary large number in the header of the video?

[scotts]

@FredrikNoren, I think that may work for now. But you'll also need to do the same for the number of frames. Please let know if it succeeds or fails!

I'd like to learn some more about your use-case. Are you basically just grabbing the next N frames that you can on the live stream? Are you ever seeking backward, or seeking forward more than 1 frame?

[FredrikNoren]

@scotts I experimented with setting the number of frames for a while, but then I realized I can just use seek="exact" and re-open the same file over and over instead, so I figured I'd at least start there to get the ball rolling on my side. I'm guessing it will grow slow as the file video file grows larger though. Here's the code I use now:

def agent_loop(video_path: Path, actions_path: Path):
    print("Starting ml agent...")
    with actions_path.open("a") as actions_file:
        while True:
            try:
                start_time = time.time()
                dec = VideoDecoder(video_path)
                actions = run_agent_policy(dec)
                actions_file.write(json.dumps(actions) + "\n")
                actions_file.flush()
                print(f"Read frames. shape={dec[len(dec)-1].shape} time={time.time() - start_time}") # type: ignore
            except Exception as e:
                print(f"Decode video error: {e}")
                time.sleep(10)

For our use case: We'll be doing research on agents in games so we'll want to experiment with different setups. Some of them that come to mind are:

• Just have a regular policy which reads the last N frames and predicts the action, and try to have it small enough that it can run in real time
• Have a "slow" and "fast" policy, where the slow policy runs every at some fixed time interval, and the "fast" policy runs in real-time, with the input from the slow policy as conditioning
• I could also see some kind of auto-regressive policy which grabs whatever the latest frame is, and appends it to an action queue and a latent state queue; so basically it would just always grab the latest frame.

To answer your questions specifically:

• Are you basically just grabbing the next N frames that you can on the live stream?
  • I think that would be the most common use case yeah
• Are you ever seeking backward, or seeking forward more than 1 frame?
  • I could see us wanting to grab like the last N frames, but then as we go further back in time it becomes more sparse; so after N1 frames we grab every 10 frames, then every 100 frames.
  • For forward seeking; I think if we can keep the same VideoDecoder open (which would probably be ideal, so it only needs to parse the header once), then I'd imagine that we'd seek forward to the end of the stream at each loop, and then work backwards from there.