Upgrade DataFusion to version 45

[patchwork01]

Background

The following Dependabot upgrade failed for a couple of reasons:

• Bump datafusion from 44.0.0 to 45.0.0 in /rust #4231

The Arrow version didn't match the new version in DataFusion, and the Java integration test was failing because the output file from the compaction is not readable from Java. It looks like the schema in the output file has the row keys changed to optional. This test was passing before the DataFusion upgrade.

Description

We'd like to upgrade DataFusion to version 45.

Analysis

It's not clear what may have caused the schema change from the changelog for DataFusion:
https://github.com/apache/datafusion/blob/main/dev/changelog/45.0.0.md

[m09526]

Row keys in schema, shouldn't be marked as optional/nullable. Sleeper doesnt't support nullable row keys or sort keys.

[gaffer01]

But what's changed in DataFusion 45 to cause this to break now?

[m09526]

The SketchUDF column (row key 0) is now being marked as optional, instead of required. Disabling the UDF avoids the bug. Why is the "is_nullable" output no longer being used properly?

[m09526]

This is the cause:

https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.is_nullable

The "is_nullable" function in the ScalarUDFImpl has been deprecated in favour of "return_type_from_args".

Since is_nullable is never called in DataFusion 45.0.0, our sketch column is now nullable again.

Change introduced by: apache/datafusion#14094