Better support for shuffles

[razvan]

This is a dump of a short discussion.

Refine the questions below:

• Do we support assigning a volume for spilling to on the Spark pods?
• Do we support encrypted shuffles? This ensures data shuffled across the network is not sent in the clear.
• There’s mention of an external shuffle service in the Spark on K8s docs, is this something we need to support?

[razvan]

Closing this for now, as Spark on Kubernetes still doesn't support a shuffle service and there is no crear path for us to improve things here.