bug with array struct field when copying from duckdb to vortex

[uwemaurer]

What happened?

I have this small (1kb) data file which causes a crash when I try to export it to vortex.

D load vortex;
D describe select * from 'test.json';
┌─────────────┬───────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │                            column_type                            │  null   │   key   │ default │  extra  │
│   varchar   │                              varchar                              │ varchar │ varchar │ varchar │ varchar │
├─────────────┼───────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┤
│ stats       │ STRUCT(v DOUBLE, country STRUCT("key" VARCHAR, "value" DOUBLE)[]) │ YES     │ NULL    │ NULL    │ NULL    │
└─────────────┴───────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘
D copy ( select * from 'test.json') to 'test.vortex' (format vortex);
memory allocation of 2012518595690544 bytes failed
Aborted (core dumped)

test.json

Steps to reproduce

see above

Environment

• vortex extension: dae36cd
• duckdb 1.4.3

Additional context

No response

[connortsui20]

TLDR; I can repro this for the current Vortex release that this repo is on (Vortex v0.56.0), but this is also fixed when I bump Vortex to v0.57.0

On v0.56.0:

❯ RUST_BACKTRACE=1 ./build/debug/duckdb -unsigned
duckdb(88299,0x2070ee240) malloc: nano zone abandoned due to inability to reserve vm space.
DuckDB v1.4.3 (Andium) d1dc88f950
Enter ".help" for usage hints.
D LOAD json;
D LOAD vortex;
D SELECT * FROM 'test.json';
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                      stats                                                                       │
│                                        struct(v double, country struct("key" varchar, "value" double)[])                                         │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ {'v': 0.0, 'country': NULL}                                                                                                                      │
│ {'v': 0.0, 'country': NULL}                                                                                                                      │
│ {'v': 1.0, 'country': NULL}                                                                                                                      │
│ {'v': 0.0, 'country': [{'key': de, 'value': 0.007381587704121518}, {'key': no, 'value': 0.007381587704121518}, {'key': hk, 'value': 0.02458737…  │
│ {'v': 0.0, 'country': [{'key': it, 'value': 0.08628905985328311}, {'key': es, 'value': 0.014450343988245412}, {'key': br, 'value': 0.024994306…  │
│ {'v': 0.0, 'country': NULL}                                                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
D COPY (SELECT * FROM 'test.json') TO 'test.vortex' (FORMAT VORTEX);


thread '<unnamed>' panicked at /Users/connor/spiral/vortex-data/duckdb-vortex/vortex/vortex-error/src/lib.rs:346:33:
offset must fit i64: out of range integral type conversion attempted
Backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2: std::backtrace::Backtrace::create
             at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/backtrace.rs:331:13
   3: <vortex_error::VortexError as core::convert::From<core::num::error::TryFromIntError>>::from
             at ./vortex/vortex-error/src/lib.rs:540:49
   4: <T as core::convert::Into<U>>::into
             at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/convert/mod.rs:768:9
   5: <core::result::Result<T,E> as vortex_error::VortexExpect>::vortex_expect::{{closure}}
             at ./vortex/vortex-error/src/lib.rs:345:32
   6: core::result::Result<T,E>::map_err
             at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:914:27
   7: <core::result::Result<T,E> as vortex_error::VortexExpect>::vortex_expect
             at ./vortex/vortex-error/src/lib.rs:345:14
   8: vortex_duckdb::convert::vector::flat_vector_to_vortex
             at ./vortex/vortex-duckdb/src/convert/vector.rs:210:48
   9: vortex_duckdb::convert::vector::flat_vector_to_vortex::{{closure}}
             at ./vortex/vortex-duckdb/src/convert/vector.rs:236:28
...
  45: vortex_duckdb::convert::vector::data_chunk_to_vortex
             at ./vortex/vortex-duckdb/src/convert/vector.rs:258:10
  46: <vortex_duckdb::copy::VortexCopyFunction as vortex_duckdb::duckdb::copy_function::CopyFunction>::copy_to_sink
             at ./vortex/vortex-duckdb/src/copy.rs:77:21
  47: vortex_duckdb::duckdb::copy_function::callback::copy_to_sink_callback::{{closure}}
             at ./vortex/vortex-duckdb/src/duckdb/copy_function/callback.rs:89:9
  48: vortex_duckdb::duckdb::try_or
             at ./vortex/vortex-duckdb/src/duckdb/mod.rs:78:11
  49: vortex_duckdb::duckdb::copy_function::callback::copy_to_sink_callback
             at ./vortex/vortex-duckdb/src/duckdb/copy_function/callback.rs:88:5
  50: _ZN6vortex14c_copy_to_sinkERN6duckdb16ExecutionContextERNS0_12FunctionDataERNS0_18GlobalFunctionDataERNS0_17LocalFunctionDataERNS0_9DataChunkE
             at ./vortex/vortex-duckdb/cpp/copy_function.cpp:99:5
  51: _ZNK6duckdb18PhysicalCopyToFile4SinkERNS_16ExecutionContextERNS_9DataChunkERNS_17OperatorSinkInputE
             at ./duckdb/src/execution/operator/persistent/physical_copy_to_file.cpp:568:3

On later versions there is no error (except on the current version of develop which is a separate issue). I don't know the exact change that would have fixed this but I do remember seeing something similar to this getting fixed so I'll say this is solved? Or at least it will be solved when we bump the Vortex version.

[connortsui20]

This is fixed by Vortex version 0.57.0 (the next version).

[connortsui20]

Whoops I forgot that we allow creating issues for bugs again, sorry for moving this over!

[uwemaurer]

It would be great to also update and release the duckdb extension together with a vortex release. I see it is still at version 0.56.0, but vortex has released 0.58.0 already.

[connortsui20]

Hi, we are talking with the DuckDB people and it should happen soon (hopefully the next DuckDB release)!