Improve process for making changes to both arrow-datafusion and arrow-ballista

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As a follow-on to #2582 it would be nice if we could make the process a little easier and more automated.

Describe the solution you'd like
Perhaps we can implement a mechanism where we can say "Depends on ballista branch X" in the PR description and have the scripts pull that version?

Describe alternatives you've considered
Live with the current solution.

Additional context
None

[tustvold]

I've created #2632 to try to improve the workflow for arrow upgrades, as they're every two weeks. That said I feel something is a bit off with this, in my opinion Ballista should follow DataFusion upgrades, not lead them.

The current approach effectively puts the burden of Ballista maintenance onto the DataFusion maintainers, which feels somewhat contrary to the ideals of splitting it out in the first place.

Perhaps in lieu of releasing DataFusion more frequently, we could simply update the git pin within Ballista more frequently. This is what we do with IOx, and often prior to a more risky DataFusion PR we will create a candidate IOx PR to test it out. I think this strikes a pragmatic balance, leaving reviewer discretion as to when to upgrade what.

I think when a non-trivial breaking change is made to DataFusion, it is fair to expect that the contributor will help out with the corresponding changes to Ballista, but I'm not sure mandating the Ballista change is prepared in advance strikes the right balance, especially whilst the APIs of both arrow-rs and DataFusion are so in flux.

I don't know, I would appreciate your thoughts

[andygrove]

I think we can close this now. The current approach seems to be working well enough. Feel free to re-open is I am missing anything.