Can't use arrow datatype `Duration` with `datafusion 46`

[emilk]

Problem

Datafusion 46 does not support the Duration datatype:

Internal error: Min/Max accumulator not implemented for type Duration(Nanosecond).

This is fixed on Datafusion trunk:

• Implement GroupsAccumulator for min/max Duration apache/datafusion#15322

Which will be released maybe mid April (1-3 weeks away):

• Release DataFusion 47.0.0 (April 2025) apache/datafusion#15072

Where are we using datafusion

We're not, not directly, but lance is, and it's using datafusion 45.0.0

Solutions

Wait

Wait for a new datafusion release:

• Release DataFusion 47.0.0 (April 2025) apache/datafusion#15072

…and then wait for lance to update. This may take a while.

Avoid Duration datatype

We could avoid using the Duration datatype until the next datafusion/lance releases.

If we do, we should wait with the Rerun 0.23 release until then as well, which may well be a month away.

Patch datafusion in dataplatform

Use a patched datafusion version (e.g. on our own fork) in crates.io.

We could still hit the problem when using da an old datafusion from Python though.