duckdb-rrd

A DuckDB extension for querying and writing Rerun .rrd recording files with SQL.

LOAD rrd;

SELECT frame, "/metrics/loss:value"[1] AS loss
FROM read_rrd('recording.rrd', entity => '/metrics/**', timeline => 'frame')
WHERE frame > 500;

• Read .rrd recordings as tables: one row per time point, one column per entity/component, with real DuckDB types (DOUBLE[], FLOAT[3][], TIMESTAMP, nested STRUCTs, ...).
• Live: query a recording while another process is still writing it — each scan re-opens the file and tolerates a partially-written tail, so re-running a query sees the newest data. No locks are taken.
• Write: COPY (SELECT ...) TO 'out.rrd' (FORMAT rrd, ...) exports query results as a valid Rerun recording that the Rerun viewer and CLI open natively.
• Everything composes with plain SQL: JOIN across recordings, GROUP BY, window functions, other DuckDB extensions.

Built on Rerun's own low-level format crates (re_log_encoding, re_chunk_store, re_dataframe), so decoding always matches the format spec. Not affiliated with the Rerun SDK/viewer; this extension only reads and writes the file format.

Functions

read_rrd(path, ...)

Scans a recording. The schema is derived from the data: first the index (timeline) column, then one column per entity/component named <entity_path>:<component>.

named parameter	type	description
entity	VARCHAR	entity path filter expression, e.g. '/camera/**', '/metrics/** & !/metrics/debug'
timeline	VARCHAR	index timeline (default: log_time if present, else the first timeline)
fill_latest	BOOLEAN	fill sparse cells with latest-at semantics instead of NULL
static_only	BOOLEAN	only static data, no index column
recording	VARCHAR	recording id, for files that contain several recordings

Component cells are lists (a Rerun row can hold N instances of a component): a scalar logged per frame arrives as DOUBLE[] with one element, a point cloud as FLOAT[3][]. Index with [1] for single-instance components.

Introspection

SELECT * FROM rrd_recordings('file.rrd'); -- application_id, recording_id
SELECT * FROM rrd_entities('file.rrd');   -- entity paths
SELECT * FROM rrd_schema('file.rrd');     -- columns, component types, datatypes, static flags

COPY ... TO (FORMAT rrd)

COPY (SELECT frame, loss, accuracy FROM training_metrics)
TO 'metrics.rrd'
(FORMAT rrd, ENTITY '/metrics', TIMELINE 'frame', COLUMNS 'frame,loss,accuracy');

option	description
ENTITY	required; entity path the components are logged to
TIMELINE	name of the index column (default 'index'); integer columns become a sequence timeline, TIMESTAMP columns a timestamp timeline; TIMELINE '' writes static data
COLUMNS	comma-separated column names, positionally. DuckDB's copy C API exposes column types but not names to extensions, so restate them to keep them; otherwise column 0 is the index and the rest are col_1..col_N

Every column that is not the index becomes a component on ENTITY. LIST columns write one component batch per row (multi-instance); scalar columns write single-instance batches. Each COPY creates a fresh recording file.

Live tailing

An .rrd file is an append-only chunk stream, and read_rrd decodes as much of it as is complete. While a writer (e.g. rr.save(...) in a training script) holds the file open:

SELECT count(*), max(step) FROM read_rrd('live.rrd', timeline => 'step');
-- run it again a second later: more rows

Polling from SQL, dashboards, or a notebook therefore works out of the box.

Installation

Not yet published to the community extension repository, so build from source (see below), then:

-- start duckdb with: duckdb -unsigned
LOAD 'build/release/extension/rrd/rrd.duckdb_extension';

Building

Requires Rust (edition 2024 toolchain), Python 3, and GNU make.

git clone --recurse-submodules https://github.com/VertexStudio/duckdb-rrd
cd duckdb-rrd
make configure
make release        # or: make debug
make test_release   # sqllogictests, or: make test_debug

The extension lands in build/<profile>/extension/rrd/rrd.duckdb_extension, built against DuckDB v1.5.4 (see TARGET_DUCKDB_VERSION in the Makefile).

Test fixtures under test/data/ are generated by cargo run --bin rrd_testgen -- test/data.

How it works

• The file is decoded with Rerun's own crates and queried through re_dataframe::QueryEngine; read_rrd streams its record batches into DuckDB vectors.
• Rerun and duckdb-rs track different arrow major versions; arrays cross the boundary zero-copy via the Arrow C data interface (src/arrow_bridge.rs).
• The writer registers a COPY function through DuckDB's C API directly (duckdb-rs does not wrap it yet, src/write.rs), exports each sunk chunk through DuckDB's native Arrow export, and encodes generic Rerun component chunks with re_log_encoding.
• third_party/duckdb vendors the duckdb-rs crate with a one-line dependency relaxation (see third_party/README.md).

Limitations

• Blueprint stores inside .rrd files are skipped; files holding multiple recordings need recording => '...'.
• No projection/filter pushdown into the Rerun query yet — WHERE and column selection happen in DuckDB after decoding (use entity => to narrow the scan).
• COPY TO creates a new recording; appending to an existing file is not supported.
• Written component values round-trip through DuckDB types; Rerun archetype-aware exports (e.g. logging real Points3D) are not attempted.

License

MIT