Release DataFusion 42.1.0

[alamb]

Is your feature request related to a problem or challenge?

There are a few bugs introduced as part of 42.0.0 that were hit by downstream consumers

For example, @rtyler and @ion-elgreco have hit #12687 I think

delta-io/delta-rs#2886 (comment)

Describe the solution you'd like

Consider releasing 42.1.0 patch release with some code fixes. Ideas:

• Field Metadata Lost on COUNT DISTINCT queries resulting in Internal Error: Physical input schema should be the same as the one converted from logical input schema #12687
• Field Metadata Lost on CROSS JOIN #12734
• Field Metadata Lost on UNION #12735
• physical-plan: Cast nested group values back to dictionary if necessary #12586

Describe alternatives you've considered

We could also just wait for the next release

• Release DataFusion 43.0.0 #12470

Additional context

No response

[alamb]

@andygrove i wonder what your thoughts are on this idea?

[andygrove]

Sounds good to me

[brancz]

Any chance we could include #12586 ?

[andygrove]

I can start looking at this today. Is there still interest in the idea of maintaining a LTS version of DataFusion? Would 42 be a good foundation for that?

[alamb]

Is there still interest in the idea of maintaining a LTS version of DataFusion? Would 42 be a good foundation for that?

I can help with this, but I am already stretched pretty thin with other things

In terms of creating an LTS branch, perhaps we can target 43 of 44 (where I am hoping we'll be able to complete

• [DISCUSSION] Make DataFusion the fastest engine for querying parquet data in ClickBench #12821
• [Epic] Unify WindowFunction Interface (remove built in list of BuiltInWindowFunction s) #8709
• [EPIC] Automatically generate all function documentation from code #12740

[alamb]

I have created the https://github.com/apache/datafusion/tree/maint-42.x branch

It would be super helpful if someone could make cherry-pick PR(s) with the changes mentioned above targeting that branch (I promise to review them quickly!)

[andygrove]

I have created the https://github.com/apache/datafusion/tree/maint-42.x branch

It would be super helpful if someone could make cherry-pick PR(s) with the changes mentioned above targeting that branch (I promise to review them quickly!)

We do already have the https://github.com/apache/datafusion/tree/branch-42 branch

[andygrove]

Here is some info on the current branching policy: https://github.com/apache/datafusion/tree/main/dev/release#branching-policy

[alamb]

We do already have the https://github.com/apache/datafusion/tree/branch-42 branch

Sorry - I have removed the maint-42 branch

Let's make the PRs to the branch-42 branch then!

[alamb]

Since no one made any backport PRs I think we are moving on to just releasing 43.0.0 #12470 -- if anyone is interested in helping do maintenance releases we can considering doing them, but otherwise I think we only have capacity to do releases off main

[matthewmturner]

I should have picked this up - apologies.