-
Notifications
You must be signed in to change notification settings - Fork 1k
Closed
Labels
Description
Describe the bug
The force_validate check added in #1546 revealed that the UnionArrays produced from the filter kernel fails validation.
It is not clear if the validation is incorrect or the filter kernel is incorrect
To Reproduce
Uncomment the
#[cfg(not(feature = "force_validate"))]And then run the test like:
cargo test --features=force_validate -p arrowlines for the following tests and the will fail:
---- compute::kernels::filter::tests::test_filter_union_array_sparse stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34
---- compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34
Expected behavior
The tests should pass when force_validate is on
Additional context
Add any other context about the problem here.