Bug: Lance v2.1 fails on Public BI CMSProvider dataset #4943
Description
Hello!
I am one of the Vortex maintainers. I recently bumped Lance from v2.0 to v2.1 in our benchmarks (vortex-data/vortex#4930) and it caused one of our decompression benchmarks that uses the Public BI datasets from CWI (https://github.com/cwida/public_bi_benchmark) to crash.
More specifically, when we write the CMSProvider dataset with Dataset::write and WriteParams::with_storage_version(LanceFileVersion::V2_1), we get a panic. When using WriteParams::default(), there was never a crash.
We depend on Lance 0.38.2, and the same issue happens on 0.38.1 (I didn't check other versions).
We can replicate by running this command in the Vortex repository (not using release will take forever):
CARGO_PROFILE_RELEASE_DEBUG=true \
RUST_BACKTRACE=1 \
cargo r --release --bin compress -- --datasets CMSprovider --formats lance
Here is a backtrace:
Details
thread 'tokio-runtime-worker' panicked at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/repdef.rs:1407:52:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
1: core::panicking::panic_fmt
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
2: core::panicking::panic
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:145:5
3: core::option::unwrap_failed
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/option.rs:2072:5
4: core::option::Option<T>::unwrap
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:1005:21
5: lance_encoding::repdef::RepDefUnraveler::unravel_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/repdef.rs:1407:52
6: lance_encoding::repdef::CompositeRepDefUnraveler::unravel_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/repdef.rs:1494:27
7: lance_encoding::encodings::logical::primitive::StructuralCompositeDecodeArrayTask::restore_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/primitive.rs:3121:34
8: <lance_encoding::encodings::logical::primitive::StructuralCompositeDecodeArrayTask as lance_encoding::decoder::StructuralDecodeArrayTask>::decode
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/primitive.rs:3162:21
9: <lance_encoding::encodings::logical::struct::RepDefStructDecodeTask as lance_encoding::decoder::StructuralDecodeArrayTask>::decode::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/struct.rs:340:30
10: core::iter::adapters::map::map_try_fold::{{closure}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:95:28
11: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/into_iter.rs:351:25
12: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:121:19
13: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:192:14
14: core::iter::traits::iterator::Iterator::try_for_each
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2487:14
15: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:174:14
16: alloc::vec::Vec<T,A>::extend_desugared
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3578:44
17: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/spec_extend.rs:19:14
18: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:42:9
19: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter{{reify.shim}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:19:5
20: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/in_place_collect.rs:246:9
21: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3470:9
22: core::iter::traits::iterator::Iterator::collect
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2027:9
23: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2050:51
24: core::iter::adapters::try_process
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:160:17
25: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2050:9
26: core::iter::traits::iterator::Iterator::collect
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2027:9
27: <lance_encoding::encodings::logical::struct::RepDefStructDecodeTask as lance_encoding::decoder::StructuralDecodeArrayTask>::decode
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/struct.rs:341:14
28: <alloc::boxed::Box<dyn lance_encoding::decoder::StructuralDecodeArrayTask> as lance_encoding::decoder::DecodeArrayTask>::decode
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/decoder.rs:2418:9
29: lance_encoding::decoder::NextDecodeTask::into_batch
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/decoder.rs:2440:36
30: lance_encoding::decoder::StructuralBatchDecodeStream::into_stream::{{closure}}::{{closure}}::{{closure}}::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/decoder.rs:1762:31
31: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
32: <F as futures_core::future::TryFuture>::try_poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:14
33: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:31
34: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:44
35: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
36: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
37: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
38: <futures_util::stream::futures_ordered::OrderWrapper<T> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:56:29
39: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528:24
40: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
41: <futures_util::stream::futures_ordered::FuturesOrdered<Fut> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:195:49
42: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
43: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/next.rs:32:21
44: lance_table::utils::stream::MergeStream::emit::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-table-0.38.2/src/utils/stream.rs:44:41
45: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
46: lance_table::utils::stream::wrap_with_row_id_and_delete::{{closure}}::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-table-0.38.2/src/utils/stream.rs:305:38
47: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.41/src/instrument.rs:321:15
48: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:365:24
49: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/loom/std/unsafe_cell.rs:16:9
50: tokio::runtime::task::core::Core<T,S>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:354:30
51: tokio::runtime::task::harness::poll_future::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:535:30
52: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
53: std::panicking::catch_unwind::do_call
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40
54: std::panicking::catch_unwind
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19
55: std::panic::catch_unwind
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:359:14
56: tokio::runtime::task::harness::poll_future
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:523:18
57: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:210:27
58: tokio::runtime::task::harness::Harness<T,S>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:155:20
59: tokio::runtime::task::raw::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:325:13
60: tokio::runtime::task::raw::RawTask::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:255:18
61: tokio::runtime::task::LocalNotified<S>::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/mod.rs:509:13
62: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:600:18
63: tokio::task::coop::with_budget
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
64: tokio::task::coop::budget
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
65: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:591:9
66: tokio::runtime::scheduler::multi_thread::worker::Context::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:539:29
67: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:504:24
68: tokio::runtime::context::scoped::Scoped<T>::set
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9
69: tokio::runtime::context::set_scheduler::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38
70: std::thread::local::LocalKey<T>::try_with
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:315:12
71: std::thread::local::LocalKey<T>::with
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/thread/local.rs:279:20
72: tokio::runtime::context::set_scheduler
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17
73: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:499:9
74: tokio::runtime::context::runtime::enter_runtime
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
75: tokio::runtime::scheduler::multi_thread::worker::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:491:5
76: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:457:45
77: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/task.rs:42:21
78: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:365:24
79: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/loom/std/unsafe_cell.rs:16:9
80: tokio::runtime::task::core::Core<T,S>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:354:30
81: tokio::runtime::task::harness::poll_future::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:535:30
82: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9
83: std::panicking::catch_unwind::do_call
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40
84: std::panicking::catch_unwind
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19
85: std::panic::catch_unwind
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:359:14
86: tokio::runtime::task::harness::poll_future
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:523:18
87: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:210:27
88: tokio::runtime::task::harness::Harness<T,S>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:155:20
89: tokio::runtime::task::raw::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:325:13
90: tokio::runtime::task::raw::RawTask::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:255:18
91: tokio::runtime::task::UnownedTask<S>::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/mod.rs:546:13
92: tokio::runtime::blocking::pool::Task::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:161:19
93: tokio::runtime::blocking::pool::Inner::run
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:516:22
94: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:474:47
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-table-0.38.2/src/utils/stream.rs:310:46:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(6678), "called `Option::unwrap()` on a `None` value", ...)
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
1: core::panicking::panic_fmt
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
2: core::result::unwrap_failed
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
3: core::result::Result<T,E>::unwrap
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:1167:23
4: lance_table::utils::stream::wrap_with_row_id_and_delete::{{closure}}::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-table-0.38.2/src/utils/stream.rs:310:46
5: <T as futures_util::fns::FnOnce1<A>>::call_once
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/fns.rs:15:9
6: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:57:75
7: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
8: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
9: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:44
10: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
11: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
12: <F as futures_core::future::TryFuture>::try_poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:14
13: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:31
14: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:44
15: <futures_util::future::future::Inspect<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
16: <futures_util::future::try_future::InspectOk<Fut,F> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
17: <core::pin::Pin<P> as core::future::future::Future>::poll
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:124:9
18: <F as futures_core::future::TryFuture>::try_poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:14
19: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:31
20: <futures_util::stream::futures_ordered::OrderWrapper<T> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:56:29
21: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528:24
22: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
23: <futures_util::stream::futures_ordered::FuturesOrdered<Fut> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/futures_ordered.rs:195:49
24: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
25: <futures_util::stream::try_stream::try_buffered::TryBuffered<St> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_buffered.rs:63:38
26: <S as futures_core::stream::TryStream>::try_poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:14
27: <futures_util::stream::try_stream::try_filter_map::TryFilterMap<St,Fut,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_filter_map.rs:76:68
28: <S as futures_core::stream::TryStream>::try_poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:14
29: <futures_util::stream::try_stream::into_stream::IntoStream<St> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/into_stream.rs:38:31
30: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:47
31: <futures_util::stream::stream::Inspect<St,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:97:35
32: <futures_util::stream::try_stream::InspectOk<St,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:97:35
33: <lance_core::utils::futures::FinallyStream<S,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-core-0.38.2/src/utils/futures.rs:241:31
34: <S as futures_core::stream::TryStream>::try_poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:14
35: <futures_util::stream::try_stream::into_stream::IntoStream<St> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/into_stream.rs:38:31
36: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58:47
37: <futures_util::stream::try_stream::MapErr<St,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:97:35
38: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:33
39: <S as futures_core::stream::TryStream>::try_poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:206:14
40: <futures_util::stream::try_stream::try_flatten::TryFlatten<St> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/try_stream/try_flatten.rs:61:46
41: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-50.2.0/src/stream.rs:451:31
42: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:33
43: <lance_core::utils::futures::FinallyStream<S,F> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-core-0.38.2/src/utils/futures.rs:241:31
44: <datafusion_physical_plan::stream::RecordBatchStreamAdapter<S> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/datafusion-physical-plan-50.2.0/src/stream.rs:451:31
45: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:33
46: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130:33
47: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
48: <lance::dataset::scanner::DatasetRecordBatchStream as futures_core::stream::Stream>::poll_next
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-0.38.2/src/dataset/scanner.rs:3538:30
49: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:24
50: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/next.rs:32:21
51: bench_vortex::compress::lance::lance_decompress_read::{{closure}}
at ./bench-vortex/src/compress/lance.rs:53:50
52: bench_vortex::compress::bench::benchmark_lance_decompress::{{closure}}::{{closure}}
at ./bench-vortex/src/compress/bench.rs:324:14
53: bench_vortex::bench_run::run_with_setup::{{closure}}
at ./bench-vortex/src/bench_run.rs:34:41
54: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/park.rs:285:71
55: tokio::task::coop::with_budget
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5
56: tokio::task::coop::budget
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5
57: tokio::runtime::park::CachedParkThread::block_on
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/park.rs:285:31
58: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/blocking.rs:66:14
59: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/mod.rs:87:22
60: tokio::runtime::context::runtime::enter_runtime
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16
61: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
62: tokio::runtime::runtime::Runtime::block_on_inner
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:358:50
63: tokio::runtime::runtime::Runtime::block_on
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/runtime.rs:330:18
64: bench_vortex::bench_run::run_with_setup
at ./bench-vortex/src/bench_run.rs:32:31
65: bench_vortex::bench_run::run
at ./bench-vortex/src/bench_run.rs:15:5
66: bench_vortex::compress::bench::benchmark_lance_decompress
at ./bench-vortex/src/compress/bench.rs:322:16
67: compress::benchmark_compress
at ./bench-vortex/src/bin/compress.rs:263:25
68: compress::compress::{{closure}}
at ./bench-vortex/src/bin/compress.rs:141:13
69: core::iter::adapters::map::map_try_fold::{{closure}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:95:28
70: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/into_iter.rs:351:25
71: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:121:19
72: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:192:14
73: core::iter::traits::iterator::Iterator::try_for_each
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2487:14
74: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:174:14
75: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:25:41
76: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter{{reify.shim}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/spec_from_iter_nested.rs:19:5
77: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/in_place_collect.rs:246:9
78: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:3470:9
79: core::iter::traits::iterator::Iterator::collect
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2027:9
80: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2050:51
81: core::iter::adapters::try_process
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/adapters/mod.rs:160:17
82: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2050:9
83: core::iter::traits::iterator::Iterator::collect
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2027:9
84: itertools::Itertools::try_collect
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/itertools-0.14.0/src/lib.rs:2383:14
85: compress::compress
at ./bench-vortex/src/bin/compress.rs:150:10
86: compress::main
at ./bench-vortex/src/bin/compress.rs:71:5
87: core::ops::function::FnOnce::call_once
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
To spare some mental parsing, here is where the failure happens:
...
4: core::option::Option<T>::unwrap
at /Users/connor/.rustup/toolchains/1.89-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:1005:21
5: lance_encoding::repdef::RepDefUnraveler::unravel_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/repdef.rs:1407:52
6: lance_encoding::repdef::CompositeRepDefUnraveler::unravel_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/repdef.rs:1494:27
7: lance_encoding::encodings::logical::primitive::StructuralCompositeDecodeArrayTask::restore_validity
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/primitive.rs:3121:34
8: <lance_encoding::encodings::logical::primitive::StructuralCompositeDecodeArrayTask as lance_encoding::decoder::StructuralDecodeArrayTask>::decode
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/primitive.rs:3162:21
9: <lance_encoding::encodings::logical::struct::RepDefStructDecodeTask as lance_encoding::decoder::StructuralDecodeArrayTask>::decode::{{closure}}
at /Users/connor/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-encoding-0.38.2/src/encodings/logical/struct.rs:340:30
...
Note that we do not run these benchmarks on every public BI dataset (we only use these ones). So there could be other things that fail?
CC @westonpace (also thanks for the advice!)