Video stream changes to support out of order samples #12277

IsseW · 2025-12-18T18:44:35Z

What

Support out of order samples for video streams.

Implementation details

We can pre-allocate chunk samples from the rrd manifest, which are then filled in as that chunk arrives.
We no longer keep track of GOPs the same way, and no longer rely on stable gop indices. Instead we keep track of keyframes, and instead of storing indices into this we keep it sorted and binary search keyframes.
Which chunk and buffer a frame should use is now stored within the frame itself.

github-actions · 2025-12-18T18:45:03Z

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	`2ce66fe`	https://rerun.io/viewer/pr/12277	`+nightly` `+main`

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

IsseW · 2026-01-07T12:55:53Z

Since our lints are failing because of re_video depending on re_core_types. I'm considering either making chunk id a generic parameter used to identify source.

Or we could consider moving re_video out of util.

Resolved by using tuid instead of chunk id

Wumpf

Reviewed everything except video_stream_cache.rs changes and feel like I need a break atm. Maybe also better if you talk me through that one.

We should now also be able to remove the chunk store hack for not compacting video stream, right? Grep for RR-3212 to see what I mean.

Wumpf · 2026-01-08T09:30:44Z

crates/store/re_data_loader/src/lerobot/datasetv3.rs

        let end_video_time = re_video::Time::from_secs(end_time, timescale);

        // Find the GOPs that contain our time range
        let start_gop = video


below got renamed to end_keyframe. This is inconsistent now. should rename this as well

Wumpf · 2026-01-08T09:35:39Z

crates/utils/re_video/src/decode/webcodecs.rs

        let first_frame_pts = video_descr
            .samples
            .front()
+            .and_then(|s| s.sample())
            .map_or(Time::ZERO, |s| s.presentation_timestamp);


not having a first_frame_pts is fine most of the time but may cause issues. We may want to come up with a different strategy of obtaining that offset 🤔
See large comment block in submit_chunk below why we need it in the first place

Wumpf · 2026-01-08T09:38:13Z

crates/utils/re_video/src/demux/mod.rs

                0 => std::time::Duration::ZERO,
                1 => {
-                    let first = self.samples.front()?;
+                    let first = self.samples.iter().find_map(|s| s.sample())?;


method documentation should point out how we handle absent samples and gaps in sample ranges

Wumpf · 2026-01-08T09:40:04Z

crates/utils/re_video/src/demux/mod.rs

-                .iter_index_range_clamped(&seg.sample_range)
-                .map(|(_idx, sample)| sample.presentation_timestamp)
+                .iter()
+                .filter_map(|sample| Some(sample.sample()?.presentation_timestamp))


doc string should be updated to mention that reserved-only samples are simply ignored

Wumpf · 2026-01-08T09:41:43Z

crates/utils/re_video/src/demux/mod.rs

+                    .get(*p)
+                    .map(|s| s.sample())
+                    .inspect(|_s| {
+                        debug_assert!(_s.is_some(), "Keyframes should always be loaded");


nit:

Suggested change

debug_assert!(_s.is_some(), "Keyframes should always be loaded");

debug_assert!(_s.is_some(), "Keyframes mentioned in the keyframe lookup list should always be loaded");

because otherwise future-me would be confused about what magic is supposed to load all keyframes in existance

Wumpf · 2026-01-08T10:12:31Z

crates/viewer/re_renderer/src/video/player.rs

+        let sample_range = video_description.keyframe_indices[keyframe_idx]
+            ..video_description
+                .keyframe_indices
+                .get(keyframe_idx + 1)
+                .copied()
+                .unwrap_or_else(|| video_description.samples.next_index());


this "give me the GOP range starting at keyframe_idx X" happens quite a bit, worth considering having a utility for that if only to name the thing we're doing here

Wumpf · 2026-01-08T10:21:12Z

crates/utils/re_video/src/demux/mod.rs

+    /// The raw buffer this sample is in.
+    pub buffer: arrow::buffer::Buffer,


I know why we arrived at this, but I don't particularly like having arrow buffer here:

it increases the ref count for that arrow buffer a lot

it drags in libarrow dependency, until now re_video didn't need any arrow at all to do its work

arrow buffer under the hood is already a byte span into a ref counted buffer, that makes the byte_span below kinda pointless. I.e. if we're using arrow buffers here we might as well only use arrow buffers

It also means that SampleMetadata is no longer really metadata but really forms the actual sample, which is what distinguished it previously from Chunk which had a copy of the data for transfer to the decoder. I figure at that point we might as well rename it to Sample then.

... so either we check again whether we can not depend on arrow here, keeping stable buffer indices around, or we go all in and drop byte_span, rename the thing and at a later point maybe even remove Chunk

Wumpf · 2026-01-08T10:23:53Z

crates/viewer/re_renderer/src/video/player.rs

+                let Some(sample) = sample.sample() else {
+                    continue;
+                };


should we distinguish unloaded here and immediately treat things as live stream if we encounter unloaded samples?

Wumpf · 2026-01-08T10:27:06Z

crates/viewer/re_viewer_context/src/cache/video_stream_cache.rs

+    }
+}
+
+struct VideoData {


what is this struct and why is it needed? Name doesn't really tell me what it is used for

Wumpf · 2026-01-08T10:28:21Z

crates/viewer/re_viewer_context/src/cache/video_stream_cache.rs

 struct VideoStreamCacheEntry {
    used_this_frame: AtomicBool,
    video_stream: Arc<RwLock<PlayableVideoStream>>,
+    known_chunk_offsets: BTreeMap<ChunkId, ChunkSamples>,


isn't this more like chunk sample ranges?

IsseW added 📺 re_viewer affects re_viewer itself include in changelog feat-video anything video decoding, player, querying, data modelling of videos etc. labels Dec 18, 2025

IsseW marked this pull request as draft December 18, 2025 18:44

IsseW added 9 commits January 7, 2026 12:34

wip fixes

beae924

track keyframes

c61c0a8

better dbg

6d0d685

working videos, as long as you don't gc

21fa3c7

fix after rebase

9c3746c

fix chunk deletions

8b78dd0

debug to warning

7febeb0

support interlacing

7753f5a

fix lints

86b40da

IsseW force-pushed the isse/out-of-order-video-stream branch from 3924ef3 to 86b40da Compare January 7, 2026 11:45

IsseW added 2 commits January 7, 2026 12:55

remove some debugging things

f570bf8

fix dependencies

d277ebb

use tuid instead of chunk id in re_video

2ce66fe

IsseW marked this pull request as ready for review January 8, 2026 08:45

IsseW requested a review from Wumpf January 8, 2026 08:45

Wumpf requested changes Jan 8, 2026

View reviewed changes

	debug_assert!(_s.is_some(), "Keyframes should always be loaded");
	debug_assert!(_s.is_some(), "Keyframes mentioned in the keyframe lookup list should always be loaded");

		/// The raw buffer this sample is in.
		pub buffer: arrow::buffer::Buffer,

Video stream changes to support out of order samples #12277

Are you sure you want to change the base?

Video stream changes to support out of order samples #12277

Uh oh!

Conversation

IsseW commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

What

Implementation details

Uh oh!

github-actions bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IsseW commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wumpf left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IsseW commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Dec 18, 2025 •

edited

Loading

IsseW commented Jan 7, 2026 •

edited

Loading