[Feature]: Add duckdb extension

[kesavkolla]

Problem / Motivation

The duckdb ecosystem is popular. There are several extensions that people have developed so that they can be accessed easily from duckdb. I believe zvec also benefit a lot from that ecosystem. If an extension is made available for duckdb and also make it wasm compatible then zvec can easily be embedded inside browser.

Proposed Solution

Alternatives Considered

No response

Affected Area

No response

[Muitamax]

To make this work inside the browser:
You must ensure:
No OS-level dependencies
No unsupported syscalls
Pure C/C++ (or Rust compiled to WASM)
Memory allocation compatible with DuckDB’s WASM runtime
DuckDB already supports:
DuckDB-Wasm
So your extension must:
Compile cleanly to WASM
Avoid native threads (unless using WASM threading support)
Avoid file system assumptions
If zvec is serious about adoption:

Priority order should be:
Native DuckDB extension (non-WASM first)
Stabilize API
Then WASM compatibility
WASM first is risky. Native first is pragmatic.

[akuligowski9]

Do you mind if I take a swing at this?

[akuligowski9]

MVP Implementation & Design Feedback Request

I've put together an initial MVP implementation in #136 — a DuckDB extension using a collection-bridge approach (zvec owns storage, DuckDB functions act as a SQL bridge). Before going further, I'd love feedback from maintainers on a few design choices:

What the MVP does (Phase 1)

4 SQL functions that bridge DuckDB to the zvec Collection API:

• zvec_create(path, schema_json) — create a collection from a JSON schema definition
• zvec_insert(path, pk, doc_json) — insert a single document with JSON-encoded fields
• zvec_search(path, field, vector, topk) — table function for vector similarity search, returns pk, score, ...forward_fields
• zvec_fetch(path, pk) — table function to fetch a document by primary key

Supports all zvec field types (scalars, dense vectors, sparse vectors, arrays), all index types (HNSW/IVF/FLAT/INVERT), and all metrics (L2/IP/COSINE). Collections are managed by a thread-safe singleton registry.

Questions for maintainers

1. Collection-bridge vs native DuckDB storage integration: The bridge approach is simpler and exposes zvec's full feature set, but collections live outside DuckDB's catalog. Is this the right tradeoff for an MVP, or would you prefer tighter integration?

2. Extension location: Currently at duckdb_extension/ inside the zvec repo. Should this live in a separate repo instead (e.g., zvec-duckdb)?

3. JSON vs structured parameters: Schema creation and document insertion use JSON strings. An alternative would be more structured DuckDB-native parameters. Any preference?

4. Scope for Phase 2: Potential additions include zvec_optimize, zvec_delete, zvec_upsert, filter expressions in search, and batch insert from DuckDB tables. What would be most valuable?

5. DuckDB version pinning: Which DuckDB release should we target? Currently designed for the DuckDB extension API with build_static_extension/build_loadable_extension.

Happy to iterate on the design based on your feedback!

[kesavkolla]

If you don't use duckdb storage then how will in-meory option work? At times duckdb databases are in-meory with temp directory for spills. If you decouple storage then one has to figure out where and how zvec storage gets integrated.

I know duckdb does support remote data over httpfs and S3 maybe that is another route. Does zvec support object storage protocols?

[feihongxu0824]

To make this work inside the browser: You must ensure: No OS-level dependencies No unsupported syscalls Pure C/C++ (or Rust compiled to WASM) Memory allocation compatible with DuckDB’s WASM runtime DuckDB already supports: DuckDB-Wasm So your extension must: Compile cleanly to WASM Avoid native threads (unless using WASM threading support) Avoid file system assumptions If zvec is serious about adoption:

Priority order should be: Native DuckDB extension (non-WASM first) Stabilize API Then WASM compatibility WASM first is risky. Native first is pragmatic.

Agreed.

[feihongxu0824]

MVP Implementation & Design Feedback Request

I've put together an initial MVP implementation in #136 — a DuckDB extension using a collection-bridge approach (zvec owns storage, DuckDB functions act as a SQL bridge). Before going further, I'd love feedback from maintainers on a few design choices:

What the MVP does (Phase 1)

4 SQL functions that bridge DuckDB to the zvec Collection API:

• zvec_create(path, schema_json) — create a collection from a JSON schema definition
• zvec_insert(path, pk, doc_json) — insert a single document with JSON-encoded fields
• zvec_search(path, field, vector, topk) — table function for vector similarity search, returns pk, score, ...forward_fields
• zvec_fetch(path, pk) — table function to fetch a document by primary key

Supports all zvec field types (scalars, dense vectors, sparse vectors, arrays), all index types (HNSW/IVF/FLAT/INVERT), and all metrics (L2/IP/COSINE). Collections are managed by a thread-safe singleton registry.

Questions for maintainers

1. Collection-bridge vs native DuckDB storage integration: The bridge approach is simpler and exposes zvec's full feature set, but collections live outside DuckDB's catalog. Is this the right tradeoff for an MVP, or would you prefer tighter integration?
2. Extension location: Currently at duckdb_extension/ inside the zvec repo. Should this live in a separate repo instead (e.g., zvec-duckdb)?
3. JSON vs structured parameters: Schema creation and document insertion use JSON strings. An alternative would be more structured DuckDB-native parameters. Any preference?
4. Scope for Phase 2: Potential additions include zvec_optimize, zvec_delete, zvec_upsert, filter expressions in search, and batch insert from DuckDB tables. What would be most valuable?
5. DuckDB version pinning: Which DuckDB release should we target? Currently designed for the DuckDB extension API with build_static_extension/build_loadable_extension.

Happy to iterate on the design based on your feedback!

Thank you for your sharing and insights. Indeed, your approach might be quicker to implement and could enable DuckDB users to leverage Zvec's capabilities in the shortest time possible. However, further discussion is needed on how to integrate with DuckDB’s execution engine and how Zvec can use Arrow as an intermediate data transport layer.

From my perspective, to align with duckdb-vss (DuckDB’s official vector similarity search extension), I’m currently leaning toward developing a DuckDB extension based directly on Zvec Core (similar to how we plan to integrate with SQLite). This MVP would likely be named duckdb-zvec—essentially a pure index library built on top of zvec-core. That said, we’re still actively exploring and researching the exact integration approach, and we warmly welcome the community to join the discussion!

[akuligowski9]

Thanks for the detailed feedback @feihongxu0824, and great points from @kesavkolla and @Muitamax as well!

I understand the preference for building on zvec Core rather than the Collection/DB layer — having DuckDB own storage while zvec provides the index algorithms makes a lot of sense for tighter integration with DuckDB's execution engine, and aligns well with how duckdb-vss works.

A few clarifying questions to help shape the direction:

1. zvec Core API surface: Looking at src/core/, it seems like the key interfaces would be around the index builders/searchers (HNSW, IVF, FLAT). Is core_interface the right entry point, or is there a specific subset of the Core API you'd recommend building against?

2. Arrow as transport layer: You mentioned Arrow as an intermediate data transport layer. Are you thinking DuckDB's native Arrow integration (since DuckDB already uses Arrow internally) to pass vectors/results between DuckDB and zvec Core without serialization overhead? Or something more like using Arrow IPC for cross-process communication?

3. Storage ownership: In the Core-based approach, would DuckDB fully own the index storage (e.g., persisting index data in DuckDB's own catalog/storage), or would zvec Core still manage its own index files on disk with DuckDB just orchestrating the operations?

4. Scope for an initial Core-based MVP: Would a minimal version that supports just HNSW + FP32 vectors + COSINE/L2/IP be a reasonable starting point to validate the integration approach before expanding to other index types?

Happy to pivot the implementation based on your guidance. The current collection-bridge PR (#136) can serve as a reference for the DuckDB extension scaffolding and type conversion patterns even if the underlying integration layer changes significantly.