Stale dependency on arrow-rs, because it also depends on duckdb-rs

[tripokey]

The latest release of this crate uses arrow 49, as of writing the latest arrow release is at version 52. Could we get a new release with bumped arrow?

[aljazerzen]

We cannot update to 52, we are tied to the version of arrow that duckdb-rs uses.

That's because this crate (actually only some of its features) depend on duckdb-rs, which currently depends on arrow 51. The code in this crate assumes that "our" arrow versions matches with the duckdb's version.

This problem is made much much worse by the versioning cadence of arrow-rs. Read more about that here.

Maybe it would be possible to have two versions of arrow in the project and then convert duckdb arrow objects into "our" objects?

[tripokey]

Maybe a release using arrow 51 would be possible?

Two arrow versions would increase compile times as well as the final binary size.

[aljazerzen]

Oh yes, release - that's easy :D

Two arrow versions would increase compile times as well as the final binary size.

Yes, probably. Although that would only duckdb connector, while current state is holding back all and each connectors.

The real solution is for arrow-rs to stop publishing major versions of crates without breaking changes. But that requires a bit of coordination and increased release management on their side.

[aljazerzen]

Even though a new version of connector_arrow is now released, let's keep this issue open because arrow versioning is a problem for this crate.

[aljazerzen]

I now also realize that updating arrow version in connector_arrow is a breaking change, as it is exposed in the public API. Which also means that the crazy release cadence of arrow-rs is also forced on connector_arrow.

Fortunately, this crate has not yet published 1.0 version, so having 0.15 in a few years will not be as damming as arrow-rs's 51.0 :D

[tripokey]

Could you make a new release now that arrow 52 has been merged? 🥺

[aljazerzen]

Done

[tripokey]

Heads up, arrow < 53.1 and crates depending on it are affected by:

Crate:     lexical-core
Version:   0.8.5
Warning:   unsound
Title:     Multiple soundness issues
Date:      2023-09-03
ID:        RUSTSEC-2023-0086
URL:       https://rustsec.org/advisories/RUSTSEC-2023-0086

https://github.com/apache/arrow-rs/blob/b7381de6c4d36a34e63886c8ff36084181ad56dd/CHANGELOG.md?plain=1#L113

[aljazerzen]

Waiting for duckdb to publish a new version with arrow 53.