Skip to content

Support compression in spill files #16130

Open
@2010YOUY01

Description

@2010YOUY01

Is your feature request related to a problem or challenge?

Part of #16065

In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.

Describe the solution you'd like

  1. Add a configuration for spill compression type
  2. Pass an option to Arrow-IPC writer/reader for compression

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions