Ballista: Finish implementing shuffle mechanism #707
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The shuffle mechanism in Ballista is not yet working and there are some design issues that need resolving in order to fix this.
When running TPC-H query 12 I see query plans like this being executed:
ShuffleWriterExec: None
ParquetExec: ...
The output from each stage is not read by the next stage. The next stage is redundantly executing the previous stages again.
Describe the solution you'd like
Get shuffles working.
Describe alternatives you've considered
None
Additional context
None