crash when using list fields

[uwemaurer]

$ RUST_BACKTRACE=1 duckdb 
DuckDB v1.4.0 (Andium) b8a06e4a22
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D load vortex;
D copy (select ['a']) to 'test.vortex' (FORMAT VORTEX);

thread '<unnamed>' panicked at vortex-duckdb/src/convert/dtype.rs:245:46:
not yet implemented
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
   2: core::panicking::panic
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:145:5
   3: <vortex_dtype::dtype::DType as vortex_duckdb::convert::dtype::FromLogicalType>::from_logical_type
   4: vortex_duckdb::duckdb::copy_function::callback::bind_callback
   5: _ZN6vortex10c_bind_oneERN6duckdb13ClientContextERNS0_21CopyFunctionBindInputERKNS0_6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEELb1EEERKNS5_INS0_11LogicalTypeELb1EEE
   6: _ZN6duckdb6Binder10BindCopyToERNS_13CopyStatementERKNS_12CopyFunctionENS_10CopyToTypeE
   7: _ZN6duckdb6Binder4BindERNS_13CopyStatementENS_10CopyToTypeE
   8: _ZN6duckdb6Binder4BindERNS_12SQLStatementE
   9: _ZN6duckdb7Planner10CreatePlanERNS_12SQLStatementE
  10: _ZN6duckdb13ClientContext31CreatePreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEE
  11: _ZN6duckdb13ClientContext23CreatePreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEENS_21PreparedStatementModeE
  12: _ZN6duckdb13ClientContext24PendingStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERKNS_22PendingQueryParametersE
  13: _ZN6duckdb13ClientContext35PendingStatementOrPreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE
  14: _ZN6duckdb13ClientContext43PendingStatementOrPreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE
  15: _ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS4_ELb1EEERKNS_22PendingQueryParametersEb
  16: _ZN6duckdb13ClientContext12PendingQueryENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS2_ELb1EEERSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIKSC_SD_EEEb
  17: _ZN6duckdb13ClientContext12PendingQueryENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS2_ELb1EEEb
  18: _ZN6duckdb10Connection12PendingQueryENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS2_ELb1EEEb
  19: duckdb_shell_sqlite3_prepare_v2
  20: _ZN12duckdb_shell10ShellState10ExecuteSQLEPKcPPc
  21: _ZN12duckdb_shell10ShellState13RunOneSqlLineENS_9InputModeEPc
  22: _ZN12duckdb_shell10ShellState12ProcessInputENS_9InputModeE
  23: main
  24: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  25: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:392:3
  26: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5, aborting
Aborted (core dumped)

[uwemaurer]

Also crashes with struct fields: copy (select {'a':1}) to 'test.vortex' (FORMAT VORTEX);

[uwemaurer]

sorry I only notice the not yet implemented now, so it is intentional. (I think failing this SQL query with an error would be nicer instead of crashing the whole process)

Are there plans to support lists / structs soon? Or maybe just a new release of the extension is needed?

[robert3005]

I have made a pr to implement missing features, the logic already existed and it was mostly a matter of plumbing the necessary api calls. FWIW this repo exists due to duckdb extension publishing setup and contains no logic beyond publish. The actual extension lives in vortex-data/vortex

[uwemaurer]

Thank you for implementing this!

It would be great to build a new release of the duckdb extension with these new features.

[uwemaurer]

@robert3005

I tried it today, but I still get a crash.

I updated to latest duckdb (1.4.1) and the latest vortex extension (0.54.0) (duckdb/community-extensions#754)

The problem is in a different location now (vector.rs) instead of dtype.rs.

please let me know if I should open a new issue for this, or this one can be reopened

$ RUST_BACKTRACE=1 duckdb 
DuckDB v1.4.1 (Andium) b390a7c376
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D update extensions;
┌────────────────┬────────────┬─────────────────────┬──────────────────┬─────────────────┐
│ extension_name │ repository │    update_result    │ previous_version │ current_version │
│    varchar     │  varchar   │       varchar       │     varchar      │     varchar     │
├────────────────┼────────────┼─────────────────────┼──────────────────┼─────────────────┤
│ vortex         │ community  │ NO_UPDATE_AVAILABLE │ 9ea6981          │ 9ea6981         │
└────────────────┴────────────┴─────────────────────┴──────────────────┴─────────────────┘
D load vortex;
D copy (select ['a']) to 'test.vortex' (FORMAT VORTEX);

thread '<unnamed>' panicked at vortex-duckdb/src/convert/vector.rs:314:14:
not yet implemented: missing impl for DUCKDB_TYPE_LIST
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
   1: core::panicking::panic_fmt
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
   2: vortex_duckdb::convert::vector::flat_vector_to_arrow_array
   3: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
   4: <vortex_duckdb::copy::VortexCopyFunction as vortex_duckdb::duckdb::copy_function::CopyFunction>::copy_to_sink
   5: vortex_duckdb::duckdb::copy_function::callback::copy_to_sink_callback
   6: _ZN6vortex14c_copy_to_sinkERN6duckdb16ExecutionContextERNS0_12FunctionDataERNS0_18GlobalFunctionDataERNS0_17LocalFunctionDataERNS0_9DataChunkE
   7: _ZNK6duckdb18PhysicalCopyToFile4SinkERNS_16ExecutionContextERNS_9DataChunkERNS_17OperatorSinkInputE
   8: _ZN6duckdb16PipelineExecutor19ExecutePushInternalERNS_9DataChunkERNS_15ExecutionBudgetEm
   9: _ZN6duckdb16PipelineExecutor7ExecuteEm
  10: _ZN6duckdb12PipelineTask11ExecuteTaskENS_17TaskExecutionModeE
  11: _ZN6duckdb12ExecutorTask7ExecuteENS_17TaskExecutionModeE
  12: _ZN6duckdb8Executor11ExecuteTaskEb
  13: _ZN6duckdb13ClientContext19ExecuteTaskInternalERNS_17ClientContextLockERNS_15BaseQueryResultEb
  14: _ZN6duckdb18PendingQueryResult15ExecuteInternalERNS_17ClientContextLockE
  15: _ZN6duckdb18PendingQueryResult7ExecuteEv
  16: duckdb_shell_sqlite3_print_duckbox
  17: _ZN12duckdb_shell10ShellState24ExecutePreparedStatementEP12sqlite3_stmt
  18: _ZN12duckdb_shell10ShellState10ExecuteSQLEPKcPPc
  19: _ZN12duckdb_shell10ShellState13RunOneSqlLineENS_9InputModeEPc
  20: _ZN12duckdb_shell10ShellState12ProcessInputENS_9InputModeE
  21: main
  22: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  23: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  24: _start

[a10y]

I've reproduced this on latest version as well.

Looking at that code, I suspect we'll fail for converting DuckDB struct arrays into Vortex as well.

[robert3005]

turns out I added the logic for reading lists and structs from vortex into duckdb but missed the part where duckdb sends us list and structs to write