[DISCUSS] Reducing cadence of major arrow-rs releases introducing patch releases

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As more people use arrow, the overall burden to users from frequent major releases is increasing. Furthermore, the pace of breaking API changes is decreasing, so the burden on maintainers to avoid breaking changes is decreasing

As the arrow crate becomes more widely used in the ecosystem by projects other than DataFusion and other early adopters, the frequent major releases causes several issues:

1. Crates must match the major arrow versions. For example, if a crate uses DataFusion that forces everything in the entire project to exactly that version of arrow-rs).
2. parquet and arrow releases are coupled so releasing a version of parquet requires releasing a new version of arrow

The major version bumps imposes non trivial overhead on user crates. Some crates like arrow_serde have implemented clever, though complex, workaround like having feature flags for each arrow version (see the recent discussion with @chmp on arrow_serde chmp/serde_arrow#131)

Also, from what I can see many of the recent arrow-rs changes aren't really adding new APIs, they are more like filling in feature gaps and bugs, which also reflected in the slower pace of the last few releases.

Describe the solution you'd like
I propose we set a more regular major release cadence (e.g. every 3 months) and only do minor, compatible, releases between those releases.

This would absolutely require more maintainer effort, but at this stage in the project the effort may be more manageable as the APIs are in a pretty good place I think

Describe alternatives you've considered
I think there are various alternatives to trigger releases / what cadence. I don't have a hugely strong opinion in this matter

Additional context
At some point in the past we actually had fewer major releases -- see #1120

There was non trivial process overhead so we (well , really I) abandoned it and went YOLO on major releases as there wasn't really any maintenance bandwidth to do anything else

[tustvold]

I have been taking a somewhat brute force approach to this, by simply reducing the cadence of releases. This isn't necessarily ideal, but it does somewhat avoid this issue. I think the key to making this work, is to devise a minimally intrusive process where we can still make breaking changes, but still maintain patch releases. I don't really know the best way to achieve this.

[Xuanwo]

I once proposed an idea named MSAV (minimum supported arrow version) on icelake-io/icelake#234.

Libraries relying on arrow should declare an MSAV and use >=47, rather than sticking to a specific version. Meanwhile, binaries should opt for a precise version of arrow to utilize.

Most libraries only use a part of arrow's public API, which is relatively stable. However, if there are changes, libraries can update their MSAV, allowing downstream users to be aware of and adapt to these changes.

Just so you know, I'm not sure if it's a good idea.

[alamb]

I think the key to making this work, is to devise a minimally intrusive process where we can still make breaking changes, but still maintain patch releases. I don't really know the best way to achieve this.

Here are some options I can think of:

1. Release all versions from main: (merge no breaking API changes to main until we are ready to make another major version, but otherwise keep the release cadence the same)
2. Release minor versions from a maintenance branch (merge anything to main, and backport compatible changes to maintenance branch)

I think option 1 is the lowest maintainer overhead approach, but has the drawbacks of:

1. Limits when breaking API changes can be merged
2. We would have to come up with some criteria of "when would we merge such breaking changes"

[alamb]

I once proposed an idea named MSAV (minimum supported arrow version) on icelake-io/icelake#234.

Thanks for the idea @Xuanwo -- it isn't clear to me how the MSAV approach would be different from having arrow-re release minor versions (e.g. 47.1.0, 47.2.0, 47.3.0)

[Xuanwo]

it isn't clear to me how the MSAV approach would be different from having arrow-re release minor versions

MSAV enables arrow to make breaking changes without requiring library developers to update their MSAV, provided the library remains unaffected.

For instance, the library sun only uses parquet v47. When Arrow releases version 48, the developer reviews the changelog and discovers that only arrow-array is affected. Therefore, they can continue using version >=47 without needing to upgrade.

And, yes. I think this approach is difficult to implement across the entire ecosystem.

[aljazerzen]

I like this proposal. In essence, it means decoupling versions of Arrow the project and arrow-rs the crate.

Even more robust way of versioning would be to only release a major version every X amount of months, but only if there were any breaking changes. Otherwise, only a minor version can be published.

---

Regarding MSAV approach: I don't think this is sound advice to users of arrow-rs - it would lead to occasional breaking builds.

That's because if, for example, library sun uses parquet >= 47 and then Arrow releases a new version, builds of exising sun could start failing, since semver requirement of >= 48 would also match 48. At this point, sun could release a new version that would state it depends on >=48, but the old version would be broken. Which means that all dependents of sun would also break.

MSRV works only because all new versions of Rust are guaranteed to be backward compatible - they never introduce breaking changes. This is noted with the fact that Rust itself is on major version 1.x for a long time now.

[alamb]

I like this proposal. In essence, it means decoupling versions of Arrow the project and arrow-rs the crate.

Thanks @aljazerzen -- can you be clearer about what proposal you are referring to?

[aljazerzen]

I propose we set a more regular major release cadence (e.g. every 3 months) and only do minor, compatible, releases between those releases.

I mean the original one.

---

I found this issue because I'm running into the problem of managing transitive dependencies to arrow-rs. Because I depend on duckdb-rs and because it had not need updated to use arrow-rs 50, I cannot upgrade my dependency on arrow-rs, since I want compatible versions.

Other popular crates (such as serde or regex) don't cause this problem because they only publish minor or patch releases. If this was the case with arrow-rs, then duckdb-rs could depend on arrow-rs~=1.49, which would be compatible with a new version of arrow-rs 1.50.

Obviously arrow-rs cannot go back to version 1.x, but it could stop releasing new major versions that don't contain breaking changes or at least release them with less cadence.

It would mean much less toll on downstream maintainers.

[alamb]

Obviously arrow-rs cannot go back to version 1.x, but it could stop releasing new major versions that don't contain breaking changes or at least release them with less cadence.

Yes, I think this is the key -- the actual version number isn't really important -- what is important is not releasing breaking changes.

It would mean much less toll on downstream maintainers.

Indeed

[alamb]

Another point that @tustvold made that is worth repeating is that in the current model, we sometimes make new APIs that are expected to be changed prior to the next breaking major release, so ensuring we don't release such an API would be an additional overhead / require some additional discipline

[tustvold]

I think we should separate two issues that appear to have gotten conflated:

1. Less frequent major releases
2. More frequent releases

I'm in favour of 1. and we could probably aim to hew closer to quarterly major releases (we're relatively close atm).

I think 2. is harder, and tbh I am not sure how many people are really calling for this. We could/should do patch releases when sufficient functionality has accumulated, but I'm less keen on committing to a regular cadence

[mbrobbel]

we sometimes make new APIs that are expected to be changed prior to the next breaking major release

What about doing something like tokio: https://docs.rs/tokio/latest/tokio/#unstable-features?

[alamb]

think 2. is harder, and tbh I am not sure how many people are really calling for this.

With my arrow-rs user (not maintainer) hat on, I woud like to call for this:

Examples: IN DataFusion we had several features sit for 2+ months downstream in DataFusion (for example apache/datafusion#8693 from @Jefffrey ) waiting on a release that contained a non breaking API change).

InfluxDB: @erratic-pattern is working on a feature internally that is waiting on #5433 (though I think that technically is a breaking API change)