Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full support for creating and writing to data chunks #85

Open
jraymakers opened this issue Dec 29, 2024 · 0 comments
Open

Full support for creating and writing to data chunks #85

jraymakers opened this issue Dec 29, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@jraymakers
Copy link
Contributor

The current (high-level) API provides very limited (barely usable) support for creating data chunks from scratc, and no support for writing data values to the chunk. This support is highly desirable, because it would allow efficiently loading arbitrary data using the appender (using DuckDBAppender.appendDataChunk).

My current plan is to extend the current DuckDBVector interfaces to allow writes, and ensure these writes make their way to the underlying vectors in DuckDB. There are some interesting challenges and tradeoffs in making these writes efficient; ideally flushing to underlying vectors could be batched, but this might mean the JS data and the DuckDB data could be out of sync for some time, which could lead to surprising behavior.

I also plan to make it easier to create data chunks from scratch, by allowing their types to be specified using DuckDBType helpers instead of low-level duckdb.LogicalType handles.

@jraymakers jraymakers added the enhancement New feature or request label Dec 29, 2024
@jraymakers jraymakers self-assigned this Dec 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant