Skip to content

[FEATURE] Enforce memory usage limitation on expand command #3759

@yuancu

Description

@yuancu

Is your feature request related to a problem?

The expand command extracts elements in an array field and explodes it into multiple rows. This may easily cause a huge memory footage. E.g. If each array include $N$ elements, and original data has $M$ rows, after expand, it will become $M \times N$ rows.

What solution would you like?
We need to devise a mechanism to allow expand set or adjust the memory usage limit.

What alternatives have you considered?
Apply the same default value for memory limits to expand.

Do you have any additional context?
Originally posted by @penghuo in #3745 (comment)

Metadata

Metadata

Assignees

Labels

PPLPiped processing languagecalcitecalcite migration releatedenhancementNew feature or request

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions