DataFusion-DuckLake

A DataFusion extension for reading and writing DuckLake catalogs. DuckLake is an integrated data lake and catalog format that stores metadata in a SQL database and data as Parquet files on disk or object storage.

The goal of this project is to make DuckLake a first-class, Arrow-native lakehouse format inside DataFusion.

This project is maintained by Hotdata with support from the community. Come talk to us on the Hotdata Discord.

• 📦 crates.io: https://crates.io/crates/datafusion-ducklake
• 📖 API docs: https://docs.rs/datafusion-ducklake
• 🧩 Feature & backend support: see COMPATIBILITY.md
• 💬 Project chat: DataFusion+DuckLake Discord — development and usage discussion
• 🧡 Meet the team: Hotdata Discord

---

Quick start

Add the crate:

cargo add datafusion-ducklake

The default build includes the statically bundled DuckDB catalog backend. Applications configure their object store implementation directly. Other catalog backends and write support are opt‑in via feature flags. See COMPATIBILITY.md for the full matrix.

# Cargo.toml — read PostgreSQL catalogs
# (to write them too, use features = ["write-postgres"])
[dependencies.datafusion-ducklake]
version = "0.7"
default-features = false
features = ["metadata-postgres", "tls-rustls-aws-lc-rs"]

metadata-postgres, multicatalog-postgres, and write-postgres do not select a TLS provider. Plain local connections work without one. For TLS, also enable one of tls-native-tls, tls-rustls-aws-lc-rs, or tls-rustls-ring on datafusion-ducklake.

The examples below also use datafusion, object_store, and url directly — add them to your [dependencies] as well (this crate does not re-export them). The write example additionally uses sqlx (with its postgres and runtime-tokio features) to open the connection pool.

Run a query against an existing PostgreSQL catalog with the bundled example:

cargo run --example basic_query --features metadata-postgres -- \
  "postgresql://user:password@localhost:5432/database" "SELECT * FROM main.users"

Configure object_store directly for local, S3, or MinIO data files. Applications can instead register another DataFusion ObjectStore, such as an OpenDAL‑backed connector.

DataFusion enables local filesystem support. S3 and MinIO applications must enable object_store/aws themselves. That feature enables Ring through its HTTP client. If another dependency enables AWS‑LC, install the intended process‑wide Rustls CryptoProvider before creating TLS clients.

(The example also accepts DuckDB, SQLite, and MySQL connection strings with the matching metadata-* feature — see COMPATIBILITY.md.)

---

Reading a catalog

Register a DuckLakeCatalog with a SessionContext and query it with normal SQL as catalog.schema.table:

use datafusion::execution::runtime_env::RuntimeEnv;
use datafusion::prelude::*;
use datafusion_ducklake::{DuckLakeCatalog, PostgresMetadataProvider};
use object_store::ObjectStore;
use object_store::aws::AmazonS3Builder;
use std::sync::Arc;
use url::Url;

// (inside an async fn)
// Read metadata from a PostgreSQL catalog
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;

// Register object stores for any non-local data (S3 / MinIO)
let runtime = Arc::new(RuntimeEnv::default());
let s3: Arc<dyn ObjectStore> = Arc::new(
    AmazonS3Builder::new()
        .with_endpoint("http://localhost:9000") // MinIO endpoint
        .with_bucket_name("ducklake-data")
        .with_access_key_id("minioadmin")
        .with_secret_access_key("minioadmin")
        .with_region("us-west-2") // any region works for MinIO
        .with_allow_http(true)    // required for http:// endpoints
        .build()?,
);
runtime.register_object_store(&Url::parse("s3://ducklake-data/")?, s3);

let catalog = DuckLakeCatalog::new(provider)?;
let ctx = SessionContext::new_with_config_rt(
    SessionConfig::new().with_default_catalog_and_schema("ducklake", "main"),
    runtime,
);
ctx.register_catalog("ducklake", Arc::new(catalog));

let df = ctx.sql("SELECT * FROM ducklake.main.my_table").await?;
df.show().await?;

---

Writing a catalog

PostgreSQL has two writers, both behind the write-postgres feature:

• PostgresSingleCatalogMetadataWriter — the standard, spec-compliant single-catalog layout. Same catalog shape as the SQLite and MySQL writers, so the catalog is readable (and writable) by other DuckLake implementations including DuckDB's ducklake extension. SQL CREATE TABLE AS SELECT and INSERT INTO both work. Prefer this one.
• PostgresMetadataWriter — the experimental multi-catalog layout described in its own section, for hosting many catalogs in one database. Library-specific, not in the DuckLake spec, and no CTAS.

use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, PostgresMetadataProvider, PostgresSingleCatalogMetadataWriter,
};
use std::sync::Arc;

// Bootstrap the standard DuckLake tables and point the catalog at its data root
let writer = PostgresSingleCatalogMetadataWriter::new_with_init(
    "postgresql://user:pass@localhost:5432/db",
).await?;
writer.set_data_path("/abs/path/to/data")?;

// CTAS and INSERT both work on this path
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), Arc::new(writer))?;
let ctx = SessionContext::new();
ctx.register_catalog("ducklake", Arc::new(catalog));
ctx.sql("CREATE TABLE ducklake.main.events AS SELECT 1 AS id").await?.collect().await?;

The multi-catalog path instead looks like this — tables are created through the writer API (no CTAS), then appended to with SQL:

use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, DuckLakeTableWriter, MulticatalogManager, MulticatalogProvider,
    PostgresMetadataWriter, initialize_multicatalog_schema,
};
use sqlx::postgres::PgPoolOptions;
use std::sync::Arc;

let pool = PgPoolOptions::new().connect("postgresql://user:pass@localhost:5432/db").await?;

// One-time: bootstrap the multi-catalog tables, then create a named catalog
initialize_multicatalog_schema(&pool).await?;
let catalog_id = MulticatalogManager::new(pool.clone()).create_catalog("my_catalog").await?;

// Create a table by writing the first batch through the table writer
let writer = Arc::new(PostgresMetadataWriter::with_pool(pool.clone(), catalog_id).await?);
writer.set_data_path("/abs/path/to/data")?;
let object_store: Arc<dyn object_store::ObjectStore> =
    Arc::new(object_store::local::LocalFileSystem::new());
let table_writer = DuckLakeTableWriter::new(writer.clone(), object_store)?;
table_writer.write_table("public", "events", &[batch]).await?; // `batch` is your RecordBatch

// Now append with SQL, reading the same catalog back through MulticatalogProvider
let provider = MulticatalogProvider::with_pool(pool.clone(), "my_catalog").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), writer)?;
let ctx = SessionContext::new();
ctx.register_catalog("ducklake", Arc::new(catalog));
ctx.sql("INSERT INTO ducklake.public.events VALUES (1, 'a')").await?.collect().await?;
ctx.sql("SELECT count(*) FROM ducklake.public.events").await?.show().await?;

Writer output is configurable (Parquet compression, row-group sizing by row count and byte size). See DuckLakeTableWriter for the writer options.

Writing to a standard, single-catalog DuckLake store (the spec-compliant layout) is supported today for SQLite via SqliteMetadataWriter (feature write-sqlite) and for PostgreSQL via PostgresSingleCatalogMetadataWriter (feature write-postgres), where SQL CREATE TABLE AS SELECT and INSERT INTO both work. See tests/it/sql_write_tests.rs and tests/it/postgres_single_catalog_write_tests.rs.

---

Partitioning

Partition a table by one or more columns (optionally through a transform) so that queries filtering on a partition column skip whole files:

// Declare the partition scheme *before* loading data, then INSERT as usual.
execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET PARTITIONED BY (region, year(sale_date))",
)
.await?;

Writes split rows into one Parquet file per partition value; reads then prune non-matching files automatically. Supported transforms are identity (the raw value) and year/month/day/hour; pruning currently applies to identity and year (month/day/hour are recorded but not yet used to skip files). Partitioned writes work on every writable backend — SQL INSERT/UPDATE, the low-level write entry points, the streaming session, compaction, and promote all honour the live spec — and read + pruning work on all backends. RESET PARTITIONED BY turns it off. See COMPATIBILITY.md and tests/it/partition_write_tests.rs.

---

Sort order

Order the rows inside each written file so that per-file statistics stay tight and range-filtered scans skip more:

execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET SORTED BY (sale_date DESC NULLS LAST)",
)
.await?;

The spec is recorded in the catalog (ducklake_sort_info / ducklake_sort_expression) and applied on insert, to UPDATE rewrites, and to compaction output; rows are sorted before the partition split, so each per-partition file stays a sorted subsequence. Bare-column keys are produced; other expressions are tolerated on read but never produced. RESET SORTED BY turns it off.

---

Multi-catalog (PostgreSQL, experimental)

A single PostgreSQL metadata store can host multiple independent DuckLake catalogs — useful for multi-tenant deployments or keeping many logical lakehouses in one database.

⚠️ Experimental and library-specific. This multi-catalog layout is not part of the DuckLake specification and is not (yet) supported or accepted upstream. Catalogs written this way are read back only through this crate's MulticatalogProvider — they are not interchangeable with a standard single-catalog DuckLake store. The API and on-disk/in-catalog layout may change, so treat it as a preview. PostgreSQL writes no longer require this path — use PostgresSingleCatalogMetadataWriter for the spec-compliant layout.

• Create and manage catalogs with MulticatalogManager (feature write-postgres): initialize_multicatalog_schema bootstraps the shared tables, then create_catalog, and drop_table_in_catalog manage their contents.
• Read a specific catalog with MulticatalogProvider::with_pool(pool, "name") (feature multicatalog-postgres), which plugs into a DuckLakeCatalog like any other metadata provider.

See examples/multicatalog_write.rs for an end-to-end walkthrough (bootstrap → create catalogs → write → read back).

---

Maintenance

The maintenance API handles lakehouse upkeep from Rust: expiring old snapshots, cleaning up superseded files, and reclaiming orphaned files. The concrete entry points are backend-gated (write-sqlite / write-postgres). DROP TABLE is available through MetadataWriter. See examples/maintenance_demo.rs and examples/orphan_cleanup_demo.rs.

Compaction

Two explicit, triggered operations on DuckLakeTable rewrite a table's data files into a better physical layout without changing its logical rows:

• merge_adjacent_files(state, MergeOptions) coalesces several small files (of the same schema version) into fewer larger ones. A merged file spanning multiple origin snapshots is written as a DuckLake partial data file (preserving each row's original rowid and origin snapshot), so time travel and change feeds are unaffected.
• rewrite_data_files(state, RewriteOptions) rewrites a file whose deleted fraction exceeds a threshold (default 0.95), dropping its deleted rows.

Both commit atomically in one snapshot and coexist with concurrent appends; superseded files are scheduled for deletion and reclaimed later by cleanup_old_files. See examples/compaction_demo.rs.

---

Compatibility

For the full breakdown of catalog backends, object stores, types, capabilities, and current limitations, see COMPATIBILITY.md.

A few highlights worth knowing up front:

• Reads work on DuckDB, SQLite, PostgreSQL, and MySQL; writes are SQLite/PostgreSQL only.
• Object stores: local filesystem and S3-compatible (S3, MinIO).
• Snapshots can be selected through DuckLakeCatalog (by id or timestamp) or per query with ducklake_table_at; DataFusion does not support AS OF syntax.
• Table partitioning: read + file pruning on all backends; partitioned writes on every writable backend.
• Data inlined by DuckDB's ducklake extension is not read — see COMPATIBILITY.md for the COUNT(*) undercount caveat and how to avoid it.

---

Project status

This project is in alpha and evolving alongside DataFusion and DuckLake. APIs may change as core abstractions are refined. See CHANGELOG.md for release history. Feedback, issues, and contributions are welcome.