Execution error: Failed to write Vortex file: Failed to cast -1 to U16

[Omega359]

Describe the bug

During testing of my app with datafusion 50 (I created a fork or vortex's develop branch and upgraded DF to 50 in it) I am encountering an issue writing vortex files with the error

Execution error: Failed to write Vortex file: Failed to cast -1 to U16
Backtrace:
disabled backtrace

The negative # can change between runs of my integration test however it happens every time. I've yet to figure out what is disabling the backtrace but I'll keep working on it this week. I have doubts it's related to the DF upgrade and rather is an issue in vortex. If I can I'll try and create a minimal test case however I don't have a lot of spare time atm :(

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[AdamGS]

Any additional details would be appreciated, even a not-very-minimal reproduction. I'll keep an eye out for similar issues.

BTW - we have PR doing the same work here, so you can also use it.

[AdamGS]

Nny chance any of the data you were running on has a decimal data type? Specifically decimal32 or decimal64.

[Omega359]

My app uses Datafusion for the T in ETL and there are rules to transform some of the data to 'decimal(10,3)' which likely maps to decimal64 though I haven't specifically checked.

[AdamGS]

Right now DataFusion only supports decimal128/256 (I have a PR that adds support for 32/64-bit variants), I wonder if something between the it and Vortex makes a bad cast.

[Omega359]

Could be. I'll be trying to get the stack trace out of it ... I've never seen the 'disabled stacktrace' message before.

[AdamGS]

that might be datafusion's Backtrace feature? not sure if it's enabled by default. Vortex also does a similar thing (just without a feature)

[Omega359]

I finally was able to spend a bit more time on this issue and got a backtrace:

Execution error: Failed to write Vortex file: Failed to cast -62091 to I16                                                                                                                 
Backtrace:                                                                                                                                                                                                                                                                                                                                         
   0: vortex_array::arrays::primitive::compute::cast::cast::{{closure}}                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/arrays/primitive/compute/cast.rs:55:20                                                                                                                                                                                                          
   1: core::option::Option<T>::ok_or_else                                                                                                                                                                                                                                                                                                          
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:1362:25                                                                                                                                                                                                            
   2: vortex_array::arrays::primitive::compute::cast::cast                                                                                                                                                                                                                                                                                         
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/arrays/primitive/compute/cast.rs:54:39                                                                                                                                                                                                          
   3: vortex_array::arrays::primitive::compute::cast::<impl vortex_array::compute::cast::CastKernel for vortex_array::arrays::primitive::PrimitiveVTable>::cast                                                                                                                                                                                    
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/arrays/primitive/compute/cast.rs:42:37                                                                                                                                                                                                          
   4: <vortex_array::compute::cast::CastKernelAdapter<V> as vortex_array::compute::Kernel>::invoke                                                                                                                                                                                                                                                 
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/compute/cast.rs:143:12                                                                                                                                                                                                                          
   5: <vortex_array::compute::cast::Cast as vortex_array::compute::ComputeFnVTable>::invoke                                                                                                                                                                                                                                                        
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/compute/cast.rs:51:42                                                                                                                                                                                                                           
   6: vortex_array::compute::ComputeFn::invoke                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/compute/mod.rs:117:34                                                                                                                                                                                                                           
   7: vortex_array::compute::cast::cast                                                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/compute/cast.rs:27:10                                                                                                                                                                                                                           
   8: vortex_array::arrays::primitive::downcast::<impl vortex_array::arrays::primitive::PrimitiveArray>::downcast                                                                                                                                                                                                                                  
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-array/src/arrays/primitive/downcast.rs:45:27                                                                                                                                                                                                              
   9: vortex_btrblocks::temporal::compress_temporal                                                                                                                                                                                                                                                                                                
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-btrblocks/src/temporal.rs:24:30                                                                                                                                                                                                                           
  10: vortex_btrblocks::BtrBlocksCompressor::compress_canonical                                                                                                                                                                                                                                                                                    
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-btrblocks/src/lib.rs:367:28                                                                                                                                                                                                                               
  11: vortex_btrblocks::BtrBlocksCompressor::compress                                                                                                                                                                                                                                                                                              
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-btrblocks/src/lib.rs:288:14                                                                                                                                                                                                                               
  12: <vortex_btrblocks::BtrBlocksCompressor as vortex_layout::layouts::compressed::CompressorPlugin>::compress_chunk                                                                                                                                                                                                                              
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-layout/src/layouts/compressed.rs:47:14                                                                                                                                                                                                                    
  13: <alloc::sync::Arc<dyn vortex_layout::layouts::compressed::CompressorPlugin> as vortex_layout::layouts::compressed::CompressorPlugin>::compress_chunk                                                                                                                                                                                         
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-layout/src/layouts/compressed.rs:32:23                                                                                                                                                                                                                    
  14: <vortex_layout::layouts::compressed::CompressingStrategy as vortex_layout::strategy::LayoutStrategy>::write_stream::{{closure}}::{{closure}}::{{closure}}                                                                                                                                                                                    
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-layout/src/layouts/compressed.rs:141:49                                                                                                                                                                                                                   
  15: vortex_io::runtime::handle::Handle::spawn_cpu::{{closure}}                                                                                                                                                                                                                                                                                   
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-io/src/runtime/handle.rs:108:35                                                                                                                                                                                                                           
  16: core::ops::function::FnOnce::call_once{{vtable.shim}}                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:253:5                                                                                                                                                                                                        
  17: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1971:9                                                                                                                                                                                                             
  18: <vortex_io::runtime::tokio::CurrentTokioRuntime as vortex_io::runtime::Executor>::spawn_cpu::{{closure}}                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/git/checkouts/vortex-18026bca9298dce5/e54d291/vortex-io/src/runtime/tokio.rs:64:37                                                                                                                                                                                                                             
  19: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:365:24                                                                                                                                                                                                                     
  20: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut                                                                                                                                                                                                                                                                                       
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/loom/std/unsafe_cell.rs:16:9                                                                                                                                                                                                                    
  21: tokio::runtime::task::core::Core<T,S>::poll                                                                                                                                                                                                                                                                                                  
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:354:30                                                                                                                                                                                                                     
  22: tokio::runtime::task::harness::poll_future::{{closure}}                                                                                                                                                                                                                                                                                      
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:535:30                                                                                                                                                                                                                  
  23: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once                                                                                                                                                                                                                                                
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9                                                                                                                                                                                                   
  24: std::panicking::catch_unwind::do_call                                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40                                                                                                                                                                                                           
  25: std::panicking::catch_unwind                                                                                                                                                                                                                                                                                                                 
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19                                                                                                                                                                                                           
  26: std::panic::catch_unwind                                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14                                                                                                                                                                                                               
  27: tokio::runtime::task::harness::poll_future                                                                                                                                                                                                                                                                                                   
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:523:18                                                                                                                                                                                                                  
  28: tokio::runtime::task::harness::Harness<T,S>::poll_inner                                                                                                                                                                                                                                                                                      
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:210:27                                                                                                                                                                                                                  
  29: tokio::runtime::task::harness::Harness<T,S>::poll                                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:155:20                                                                                                                                                                                                                  
  30: tokio::runtime::task::raw::RawTask::poll                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:255:18                                                                                                                                                                                                                      
  31: tokio::runtime::task::LocalNotified<S>::run                                                                                                                                                                                                                                                                                                  
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/mod.rs:509:13                                                                                                                                                                                                                      
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}                                                                                                                                                                                                                                                              
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:677:22                                                                                                                                                                                                 
  33: tokio::task::coop::with_budget                                                                                                                                                                                                                                                                                                               
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:167:5                                                                                                                                                                                                                          
  34: tokio::task::coop::budget                                                                                                                                                                                                                                                                                                                    
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/task/coop/mod.rs:133:5                                                                                                                                                                                                                          
  35: tokio::runtime::scheduler::multi_thread::worker::Context::run_task                                                                                                                                                                                                                                                                           
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:591:9                                                                                                                                                                                                  
  36: tokio::runtime::scheduler::multi_thread::worker::Context::run                                                                                                                                                                                                                                                                                
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:539:29                                                                                                                                                                                                 
  37: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}                                                                                                                                                                                                                                                               
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:504:24                                                                                                                                                                                                 
  38: tokio::runtime::context::scoped::Scoped<T>::set                                                                                                                                                                                                                                                                                              
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/scoped.rs:40:9                                                                                                                                                                                                                  
  39: tokio::runtime::context::set_scheduler::{{closure}}                                                                                                                                                                                                                                                                                          
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:38                                                                                                                                                                                                                       
  40: std::thread::local::LocalKey<T>::try_with                                                                                                                                                                                                                                                                                                    
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:315:12                                                                                                                                                                                                        
  41: std::thread::local::LocalKey<T>::with                                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:279:20                                                                                                                                                                                                        
  42: tokio::runtime::context::set_scheduler                                                                                                                                                                                                                                                                                                       
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context.rs:176:17                                                                                                                                                                                                                       
  43: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:499:9                                                                                                                                                                                                  
  44: tokio::runtime::context::runtime::enter_runtime                                                                                                                                                                                                                                                                                              
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/context/runtime.rs:65:16                                                                                                                                                                                                                
  45: tokio::runtime::scheduler::multi_thread::worker::run                                                                                                                                                                                                                                                                                         
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:491:5                                                                                                                                                                                                  
  46: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}                                                                                                                                                                                                                                                                 
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/scheduler/multi_thread/worker.rs:457:45                                                                                                                                                                                                 
  47: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll                                                                                                                                                                                                                                                      
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/task.rs:42:21                                                                                                                                                                                                                  
  48: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:365:24                                                                                                                                                                                                                     
  49: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut                                                                                                                                                                                                                                                                                       
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/loom/std/unsafe_cell.rs:16:9                                                                                                                                                                                                                    
  50: tokio::runtime::task::core::Core<T,S>::poll                                                                                                                                                                                                                                                                                                  
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/core.rs:354:30                                                                                                                                                                                                                     
  51: tokio::runtime::task::harness::poll_future::{{closure}}                                                                                                                                                                                                                                                                                      
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:535:30                                                                                                                                                                                                                  
  52: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once                                                                                                                                                                                                                                                
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9                                                                                                                                                                                                   
  53: std::panicking::catch_unwind::do_call                                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40                                                                                                                                                                                                           
  54: std::panicking::catch_unwind                                                                                                                                                                                                                                                                                                                 
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19                                                                                                                                                                                                           
  55: std::panic::catch_unwind                                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14                                                                                                                                                                                                               
  56: tokio::runtime::task::harness::poll_future                                                                                                                                                                                                                                                                                                   
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:523:18                                                                                                                                                                                                                  
  57: tokio::runtime::task::harness::Harness<T,S>::poll_inner                                                                                                                                                                                                                                                                                      
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:210:27                                                                                                                                                                                                                  
  58: tokio::runtime::task::harness::Harness<T,S>::poll                                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/harness.rs:155:20                                                                                                                                                                                                                  
  59: tokio::runtime::task::raw::RawTask::poll                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/raw.rs:255:18                                                                                                                                                                                                                      
  60: tokio::runtime::task::UnownedTask<S>::run                                                                                                                                                                                                                                                                                                    
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/task/mod.rs:546:13                                                                                                                                                                                                                      
  61: tokio::runtime::blocking::pool::Task::run                                                                                                                                                                                                                                                                                                    
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:161:19                                                                                                                                                                                                                 
  62: tokio::runtime::blocking::pool::Inner::run                                                                                                                                                                                                                                                                                                   
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:516:22                                                                                                                                                                                                                 
  63: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}                                                                                                                                                                                                                                                                           
             at /home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.1/src/runtime/blocking/pool.rs:474:47                                                                                                                                                                                                                 
  64: std::sys::backtrace::__rust_begin_short_backtrace                                                                                                                                                                                                                                                                                            
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:158:18                                                                                                                                                                                                       
  65: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}                                                                                                                                                                                                                                                                             
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:559:17                                                                                                                                                                                                          
  66: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once                                                                                                                                                                                                                                                
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272:9                                                                                                                                                                                                   
  67: std::panicking::catch_unwind::do_call                                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:589:40                                                                                                                                                                                                           
  68: std::panicking::catch_unwind                                                                                                                                                                                                                                                                                                                 
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552:19                                                                                                                                                                                                           
  69: std::panic::catch_unwind                                                                                                                                                                                                                                                                                                                     
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:359:14                                                                                                                                                                                                               
  70: std::thread::Builder::spawn_unchecked_::{{closure}}                                                                                                                                                                                                                                                                                          
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:557:30                                                                                                                                                                                                          
  71: core::ops::function::FnOnce::call_once{{vtable.shim}}                                                                                                                                                                                                                                                                                        
             at /home/ubuntu/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:253:5                                                                                                                                                                                                        
  72: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once                                                                                                                                                                                                                                                                     
             at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/alloc/src/boxed.rs:1971:9                                                                                                                                                                                                                                                  
  73: std::sys::pal::unix::thread::Thread::new::thread_start                                                                                                                                                                                                                                                                                       
             at /rustc/1159e78c4747b02ef996e55082b704c09b970588/library/std/src/sys/pal/unix/thread.rs:107:17                                                                                                                                                                                                                                      
  74: start_thread                                                                                                                                                                                                                                                                                                                                 
             at ./nptl/pthread_create.c:447:8                                                                                                                                                                                                                                                                                                      
  75: clone3                                                                                                                                                                                                                                                                                                                                       
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78:0                                     

[robert3005]

I think #4735 is the fix but I am guessing from my understanding of the code

[Omega359]

After a quick update to latest release it does look like #4735 fixes it - thanks for the info! Now onto investigate increased memory usage ...