[Improvement] Support ZSTD in the spark2.x version

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

In PR of #254, the ZSTD is introduced in uniffle. But in this implementation, it only support the jobs of spark3.x .

So we should make it compatible with the spark 2.x

How should we improve?

We should do some compatibility work for the spark 2.x and 3.x ZSTD version

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[a-li]

I want to give this a shot. @zuston Could you provide me with some tips/pointers to get started? In addition, how do I go about manually testing this?

[zuston]

This problem will happen when enable ZSTD compression of uniffle in Spark 2.x, which is caused by the incompatible interface of different version ZSTD. In uniffle, ZSTD version is 1.5.2-3, but Spark 2.x built-in version is 1.3.2-2 (https://github.com/apache/spark/blob/807e0a484d1de767d1f02bd8a622da6450bdf940/pom.xml#L586).

And I think this problem could be fixed by using the reflection to be compatible with older interface of ZSTD.

Feel free to discuss more. @a-li

[capDoYeonLee]

@zuston Hi, Is this issue still valid?
I would like to try it out! could I try this on?