Introduce the zstd codec for native spill

[zuston]

Is your feature request related to a problem? Please describe.

In current codebase, the lz4 codec is used in the spill. zstd should be supported.

Additional context

I will do this if no rejection from project owner.

[richox]

i suggest using a property other than spark.io.compression.codec since it is used in broadcast/shuffle where data goes through the network. for local spilling we would like to use a lightweight compression algorithm like lz4/snappy.
i prefer a property like blaze.spill.compression.codec, what do you think?

[richox]

and have you done some benchmark using zstd spilling? it will get worse performance than lz4/snappy, if i don't understand wrong.

[zuston]

i suggest using a property other than spark.io.compression.codec since it is used in broadcast/shuffle where data goes through the network. for local spilling we would like to use a lightweight compression algorithm like lz4/snappy. i prefer a property like blaze.spill.compression.codec, what do you think?

Another option is acceptable.

and have you done some benchmark using zstd spilling? it will get worse performance than lz4/snappy, if i don't understand wrong.

Haven't. I'm still reading this part code.

[zuston]

And I think we still can reuse the IoCompressionReader/Writer . WDYT? @richox

[richox]

And I think we still can reuse the IoCompressionReader/Writer . WDYT? @richox

i'm afraid not, some spilled data is not in record batch format, for example AggExec spill row-based grouping keys, while IoCompressionWriter only accepts record batches.

[zuston]

From my sight, the IoCompressionWriter will accept the simple bufs