Skip to content

ScalarValue::try_from_array holds onto input array for Struct, Lists types #15205

@Dandandan

Description

@Dandandan

Describe the bug

ScalarValue::try_from_array are used in areas such as accumulators.
Currently arrowslice is used for Struct/Union/List types, causing accumulators holding onto the original arrays (rather than a single value), and thus higher memory usage.

To Reproduce

Create an (struct / list type) array of size n and use ScalarValue::try_from_array to convert it into a ScalarValue.

Expected behavior

Value is copied into a new Array or there would be needing efficient way of representing the scalar value.
Memory usage should be corresponding to a single value rather than the full input array.

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingperformanceMake DataFusion faster

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions