Open
Description
Is your feature request related to a problem or challenge?
Part of #16065
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
Describe the solution you'd like
- Add a configuration for spill compression type
- Pass an option to Arrow-IPC writer/reader for compression
Describe alternatives you've considered
No response
Additional context
No response