Ballista: Prep for fixing shuffle mechansim, part 1

[andygrove]

Which issue does this PR close?

Closes #737.

Query 12 iteration 0 took 29591.0 ms
+------------+-----------------+----------------+
| l_shipmode | high_line_count | low_line_count |
+------------+-----------------+----------------+
| MAIL       | 623092          | 934696         |
| SHIP       | 622964          | 934514         |
+------------+-----------------+----------------+
Query 12 avg time: 29591.04 ms

Rationale for this change

This fixes a bug introduced in #712 and also gets us closer to truly supporting shuffle.

What changes are included in this PR?

• Executors now collect meta-data from ShuffleWriterExec about the output partitions and return this information to the scheduler
• Comments are added where the current design is broken, with links to the relevant GitHub issue
• Added inputRows and outputRows metrics to ShuffleWriterExec

Are there any user-facing changes?

No

[andygrove]

If there are no objections I will go ahead and merge this tonight since it only touches Ballista files.

[alamb]

Looks good @andygrove -- I am sorry I have been focusing on arrow and DataFusion and don't have much experience with Ballista -- are there others in the Ballista community we can ask for feedback in the future so these PRs don't hang out for too long?

[andygrove]

@alamb No problem at all. Ballista is still very early and experimental (although now it is actually close to usable) so I think it has been hard for others to start contributing. My plan now is to focus on performance and scalability testing and optimizations and generally tidy things up and add documentation to make it easier for others to get involved. I think once we can demonstrate the value of Ballista then it will start to get some adoption and this will drive contributions.