Skip to content

filter produces invalid sparse UnionArrays #1547

@alamb

Description

@alamb

Describe the bug
The force_validate check added in #1546 revealed that the UnionArrays produced from the filter kernel fails validation.

It is not clear if the validation is incorrect or the filter kernel is incorrect

To Reproduce
Uncomment the

    #[cfg(not(feature = "force_validate"))]

And then run the test like:

cargo test --features=force_validate -p arrow

lines for the following tests and the will fail:

---- compute::kernels::filter::tests::test_filter_union_array_sparse stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34

---- compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34

Expected behavior
The tests should pass when force_validate is on

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow cratebug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions