feat(caches): Add DuckLakeCache implementation (do not merge) #695

devin-ai-integration · 2025-06-17T17:37:32Z

Notes from AJ (@aaronsteers)

Most of the implementation appears read to go when the below features is added.

🟥 Blocked

As of now, it appears UPDATE/MERGE are not yet supported:

upsert support duckdb/ducklake#66

Workaround found: First DELETE, then INSERT.

Any downsides to DELETE/INSERT statement combo for updating records in DuckLake? duckdb/ducklake#95

Add DuckLakeCache implementation

Summary

This PR implements a new DuckLakeCache as a subclass of DuckDBCache to support the DuckLake table format in PyAirbyte. The implementation follows the established patterns from MotherDuck cache and provides minimal configuration requirements.

Changes

New cache implementation: airbyte/caches/ducklake.py
- DuckLakeConfig class extending DuckDBConfig with DuckLake-specific fields
- DuckLakeCache class extending DuckDBCache with minimal configuration
- Sensible defaults using existing cache_dir pattern
Module exports: Updated airbyte/caches/__init__.py to export new classes
Example script: Added examples/run_faker_to_ducklake.py demonstrating usage

Configuration Parameters

metadata_connection_string: Connection string for DuckLake metadata database (defaults to sqlite:metadata.db)
data_path: Local directory for Parquet data files (defaults to data subdirectory)
catalog_name: Name for attached DuckLake catalog (defaults to ducklake_catalog)
storage_credentials: Optional dict for storage credentials (defaults to None)

Key Features

Minimal configuration: Only catalog_name required, all other fields have sensible defaults
Cache directory integration: Uses existing .cache directory pattern for default paths
DuckDB compatibility: Maintains compatibility with existing DuckDB destination pairing
Local storage focus: Uses SQLite for metadata and local directories for data storage

Usage Example

from airbyte.caches import DuckLakeCache

# Minimal configuration - only catalog_name required
cache = DuckLakeCache(catalog_name="my_ducklake_catalog")

# Full configuration example
cache = DuckLakeCache(
    metadata_connection_string="sqlite:./metadata.db",
    data_path="./ducklake_data/",
    catalog_name="my_catalog",
    schema_name="myschema",
)

Testing

✅ Example script runs successfully with minimal configuration
✅ All linting and formatting checks pass (ruff format, ruff check)
✅ Import verification test confirms proper defaults
✅ Processes 20,100 records across 3 streams in example run

Implementation Notes

Follows the same inheritance pattern as MotherDuckCache
Uses standard DuckDBSqlProcessor (no custom processor needed initially)
Provides foundation for future DuckLake ATTACH statement implementation
Maintains backward compatibility with existing cache infrastructure

Link to Devin run

https://app.devin.ai/sessions/1a262eb5a472438ba8f04088ad0b91bb

Requested by

AJ Steers (aj@airbyte.io)

…tion - Add DuckLakeCache as subclass of DuckDBCache following MotherDuck pattern - Support DuckLake table format with metadata database and Parquet storage - Provide sensible defaults using cache_dir pattern: - db_path defaults to 'ducklake.db' in cache directory - metadata_connection_string defaults to 'sqlite:metadata.db' in cache directory - data_path defaults to 'data' subdirectory in cache directory - catalog_name defaults to 'ducklake_catalog' - Enable minimal configuration requiring only catalog_name parameter - Add example script demonstrating usage with source-faker - Export DuckLakeCache and DuckLakeConfig in caches module Co-Authored-By: AJ Steers <aj@airbyte.io>

devin-ai-integration · 2025-06-17T17:37:36Z

Original prompt from AJ Steers:

Received message in Slack channel #ask-devin-ai:

@Devin - Let's add a DuckLakeCache option in PyAirbyte. Use this article for background and basic connectivity instructions. <https://motherduck.com/blog/getting-started-ducklake-table-format/|https://motherduck.com/blog/getting-started-ducklake-table-format/>

We'll build this as a sublass of DuckDBCache, with custom connectivity args. For now, we'll keep things generic and take a connection string (a subset of would go in the "attach" statement). If we can get away with using SqLite for the DB and local directories for the raw data store (instead of blob storage), let's do that. Otherwise, we'll need to configure either special creds for testing or another emulation layer like <http://Min.io|Min.io>. don't try any kind of emulation layer for now, and we'll try to keep things simple for as long as we can.

devin-ai-integration · 2025-06-17T17:37:37Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2025-06-17T17:41:55Z

PyTest Results (Fast Tests Only, No Creds)

229 tests ±0 229 ✅ ±0 3m 14s ⏱️ -10s
1 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 0b6cb53. ± Comparison against base commit 796200b.

♻️ This comment has been updated with latest results.

aaronsteers

Devin - I'm applying several changes to your PR. Wait 30 seconds and then pull the latest.

airbyte/caches/ducklake.py

aaronsteers · 2025-06-17T17:57:21Z

airbyte/caches/ducklake.py

+    def get_database_name(self) -> str:
+        """Return the name of the database."""
+        if self.db_path == ":memory:":
+            return "memory"
+
+        split_on = "\\" if "\\" in str(self.db_path) else "/"
+
+        return str(self.db_path).split(sep=split_on)[-1].split(".")[0]


I am not sure, but I believe this should just return self.catalog_name.

- Fix MyPy errors by moving path resolution to model_post_init method - Update get_database_name() to return catalog_name instead of parsing db_path - Remove unused imports and fix linting issues - Preserve AJ's documentation improvements for data_path and storage_credentials Co-Authored-By: AJ Steers <aj@airbyte.io>

airbyte/caches/ducklake.py

- Update duckdb dependency from ^1.1.0 to ^1.3.1 - Update duckdb-engine dependency from ^0.13.2 to ^0.17.0 - Add DuckLakeSqlProcessor with ATTACH statement handling - Update example script to use ducklake-data directory for verification - Verify DuckLake extension works and files land in correct directory Co-Authored-By: AJ Steers <aj@airbyte.io>

airbyte/_processors/sql/ducklake.py

github-actions · 2025-06-17T18:37:42Z

PyTest Results (Full)

291 tests ±0 277 ✅ ±0 15m 46s ⏱️ - 1m 24s
1 suites ±0 14 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 0b6cb53. ± Comparison against base commit 796200b.

♻️ This comment has been updated with latest results.

…with ducklake-data path Co-Authored-By: AJ Steers <aj@airbyte.io>

Co-Authored-By: AJ Steers <aj@airbyte.io>

aaronsteers · 2025-06-17T20:29:25Z

/fix-pr

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.
(This job requires that the PR author has "Allow edits from maintainers" enabled.)

PR auto-fix job started... Check job output.

✅ Changes applied successfully.

aaronsteers reviewed Jun 17, 2025

View reviewed changes

AJ's changes

aba725c

aaronsteers reviewed Jun 17, 2025

View reviewed changes

airbyte/caches/ducklake.py Outdated Show resolved Hide resolved

Update airbyte/caches/ducklake.py

bc8bfd4

aaronsteers reviewed Jun 17, 2025

View reviewed changes

airbyte/caches/ducklake.py Outdated Show resolved Hide resolved

aaronsteers and others added 2 commits June 17, 2025 11:11

Update airbyte/caches/ducklake.py

b08d3b8

aaronsteers reviewed Jun 17, 2025

View reviewed changes

airbyte/_processors/sql/ducklake.py Outdated Show resolved Hide resolved

Update airbyte/_processors/sql/ducklake.py

815232b

devin-ai-integration bot and others added 4 commits June 17, 2025 18:39

fix(caches): Fix DuckLake processor syntax and update example script …

d3ca797

…with ducklake-data path Co-Authored-By: AJ Steers <aj@airbyte.io>

fix(formatting): Apply Ruff auto-formatting to resolve CI lint issues

a0dd775

Co-Authored-By: AJ Steers <aj@airbyte.io>

update example script to also validate ducklake metadata

f54efcb

working - except for duckdb/ducklake#66

4e9b610

octavia-squidington-iii and others added 3 commits June 17, 2025 20:30

Auto-fix lint and format issues

3c0ec7c

add manual test steps to example script

70eac8b

add new emulated merge-update strategy: delete-then-insert

0b6cb53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(caches): Add DuckLakeCache implementation (do not merge) #695

feat(caches): Add DuckLakeCache implementation (do not merge) #695

devin-ai-integration bot commented Jun 17, 2025 •

edited by aaronsteers

Loading

Uh oh!

devin-ai-integration bot commented Jun 17, 2025

Uh oh!

devin-ai-integration bot commented Jun 17, 2025 •

edited by aaronsteers

Loading

Uh oh!

github-actions bot commented Jun 17, 2025 •

edited

Loading

Uh oh!

aaronsteers left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronsteers Jun 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025 •

edited

Loading

Uh oh!

aaronsteers commented Jun 17, 2025 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

feat(caches): Add DuckLakeCache implementation (do not merge) #695

Are you sure you want to change the base?

feat(caches): Add DuckLakeCache implementation (do not merge) #695

Conversation

devin-ai-integration bot commented Jun 17, 2025 • edited by aaronsteers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes from AJ (@aaronsteers)

🟥 Blocked

Add DuckLakeCache implementation

Summary

Changes

Configuration Parameters

Key Features

Usage Example

Testing

Implementation Notes

Link to Devin run

Requested by

Uh oh!

devin-ai-integration bot commented Jun 17, 2025

Uh oh!

devin-ai-integration bot commented Jun 17, 2025 • edited by aaronsteers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Devin AI Engineer

Uh oh!

github-actions bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Fast Tests Only, No Creds)

Uh oh!

aaronsteers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronsteers Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Full)

Uh oh!

aaronsteers commented Jun 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration bot commented Jun 17, 2025 •

edited by aaronsteers

Loading

devin-ai-integration bot commented Jun 17, 2025 •

edited by aaronsteers

Loading

github-actions bot commented Jun 17, 2025 •

edited

Loading

github-actions bot commented Jun 17, 2025 •

edited

Loading

aaronsteers commented Jun 17, 2025 •

edited by github-actions bot

Loading