Release DataFusion 37.1.0 (non breaking API release)

[alamb]

Is your feature request related to a problem or challenge?

@universalmind303 had a request for inclusion in DataFusion 37.0.0 #9682 that may miss the release: #9900 (comment)

Describe the solution you'd like

Make a 37.1.0 release with non breaking changes

Describe alternatives you've considered

No response

Additional context

No response

Potential content:

• Regression: Error in NamedStructField should be rewritten in OperatorToFunction with subquery #10029
• Regression: Coercion stopped working for coalesce on a dictionary column #9925
• Regression: group by count distinct doesn't work for timestamps with time zone #10042
• make udf structs public #9900
• Stack overflows and large_futures clippy errors on v37 #9893

Backport PRs

• Backport to 37: fix NamedStructField should be rewritten in OperatorToFunction in subquery regression #10103
• Backport to 37: fix Coercion stopped working for coalesce on a dictionary column #10104
• Backport to 37: group by count distinct doesn't work for timestamps with time zone #10105
• Backport to 37: make udf structs public #10107
• Backport to 37: Reduce DataFrame stack size and fix large futures warnings #10123

[alamb]

cc @andygrove

[alamb]

I started collecting some potential fixes for this release (basically regressions) in the description

[andygrove]

Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?

[alamb]

Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?

Thank you !

I didn't have any date in mind -- how about we try for early next week (of April 15)? I am happy to do the backport PRs -- should we make a maintenance branch? Or maybe we should use https://github.com/apache/arrow-datafusion/tree/branch-37 🤔 ?

[andygrove]

Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?

Thank you !

I didn't have any date in mind -- how about we try for early next week (of April 15)? I am happy to do the backport PRs -- should we make a maintenance branch? Or maybe we should use https://github.com/apache/arrow-datafusion/tree/branch-37 🤔 ?

That sounds good. My plan was to use branch-{version} for any maintenance releases, following the pattern used by Apache Spark., so we would release any 37.x.x releases from branch-37.

[alamb]

That sounds good. My plan was to use branch-{version} for any maintenance releases, following the pattern used by Apache Spark., so we would release any 37.x.x releases from branch-37.

Makes sense -- I will do that.

I'll plan to make some backport PRs tomorrow morning

[ion-elgreco]

Heey @alamb, when are you planning this release? :)

[alamb]

Heey @alamb, when are you planning this release? :)

@ion-elgreco I hope the end of this week -- I am going to do the backporting now

[alamb]

I made the following 4 backport PRs: #10103, #10104, #10105, #10107

I will also send a note to the dev list and on slack/discord in case anyone else wants to backport additional items

[Omega359]

Note that #9900 didn't in fact make all the udf's public - the string and unicode modules do not in fact have the modules changed to pub nor is default implemented for the functions that I spot checked.

[alamb]

Note that #9900 didn't in fact make all the udf's public - the string and unicode modules do not in fact have the modules changed to pub nor is default implemented for the functions that I spot checked.

Thank you -- moved the conversation to #10107 (comment)

[alamb]

We also added #10123 to this release, to reduce debug stack frame sizes. Thanks @sergiimk and @devinjdangelo

[alamb]

I am now beginning the process of creating a release candidate

[alamb]

Here is a PR with the changelog / version update: #10128

./dev/release/generate-changelog.py apache/arrow-datafusion 37.0.0 branch-37 > dev/changelog/37.1.0.md

[progval]

Would #9770 and #9969 be eligible for backport too?

[alamb]

Would #9770 and #9969 be eligible for backport too?

@progval -- If it was a regression or a serious bug I think it would be a good idea to backport. In this case I can't remember the exact sequence of events (as in some types were enabled, and then disabled, but I am not sure what was in the 37.0.0 release)

What exactly do you think we should backport?

[progval]

int8/int16/int32/int64 were enabled for a while. #9770 enabled for uint8/uint16/uint32/uint64, then #9969 disabled for int8/int16/uint8/uint16.

So backporting #9770 adds a non-breaking feature and a regression; and backporting #9969 immediately fixes that regression plus an existing bug.

[alamb]

I see -- so from that perspective 37.0.0 and earlier have a bug where int8/int16 bloom filters can incorrectly filter out incorrect answers.

I did some archeology and it seems like the in8/int16 bloom filter support was added in #7821 / shipped as part of https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/33.0.0.md

(aka the bug has been present for several months/releases yet)

Thus I think it would be useful to fix but I also don't think it is critical to include in 37.1.0

[alamb]

Release vote thread: https://lists.apache.org/thread/33bkbrlkqv962y0topx9rlqg19g5q2vk

[alamb]

Unfortunately it appears I made an error and forgot to update the release version in this RC

I have started a new thread[1] for a second RC https://lists.apache.org/thread/0md6qyhw0hody8p0v9wddvt7vo8r8z2x

[alamb]

The release was approved and published to crates.io: https://lists.apache.org/thread/v6y745zyljoor471964l3tfq8m37lzh2

The release is available here:
https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-37.1.0

It has also been released to crates.io: https://crates.io/crates/datafusion/37.1.0

Thank you to everyone 🙏

[ion-elgreco]

Thanks @alamb!

I tried it with 37.1.0 in delta-rs, but we still get this error: internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction, wasn't this regression fixed?

[alamb]

Thanks @alamb!

I tried it with 37.1.0 in delta-rs, but we still get this error: internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction, wasn't this regression fixed?

Hi @ion-elgreco -- another reason this could happen is that the array function's haven't been registered with your session context / however you are

Yeah you need to use the FunctionRewrtiter here (with the relevant rewriter registered) https://github.com/apache/arrow-datafusion/blob/0573f78c7e7a4d94c3204cee464b3860479e0afb/datafusion/optimizer/src/analyzer/function_rewrite.rs#L33

There was a similar question in discord here: https://discord.com/channels/885562378132000778/1166447479609376850/1229122082256851054

Can you point me at the code of how you are running your query?

[ion-elgreco]

@alamb this is the code:

  let (table, _metrics) = DeltaOps(table)
            .delete()
            .with_predicate("props['a'] = '2021-02-02'")
            .await
            .unwrap();

Which comes from here: https://github.com/delta-io/delta-rs/blob/main/crates%2Fcore%2Fsrc%2Foperations%2Fdelete.rs#L770-L774

[alamb]

Filed #10181 to track the issue with internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction