Discussion: relationship / unification of arrow-rs and arrow2 going forward

[alamb]

TLDR: please comment on this ticket if you have opinions about if and/or how the community should unite its efforts on a single Rust implementation of Apache Arrow.

Related mailing list thread: https://lists.apache.org/thread/dsyks2ylbonhs8ngnx6529dzfyfdjjzo

There is active discussion and a PR apache/datafusion#1556 about switching the DataFusion project to use the arrow2 Rust implementation of Arrow from @jorgecarleitao. While this DataFusion PR is not yet ready to merge, if DataFusion were to switch to arrow2, that leaves a question of what will happen with this (arrow-rs) code.

Since many of the PRs, contributors and maintainers of this (arrow-rs) crate are part of the DataFusion community, I believe if DataFusion switches to arrow2, much of the maintenance and extension efforts would follow arrow2

arrow2is largely developed by @jorgecarleitao, who is an Apache Arrow PMC member and committer, but the project itself has not been under the Apache Software Foundation’s governance. Additional background can be found on the mailing list archives and past mailing list threads such as this and this

It is my opinion that the Rust / Arrow / DataFusion community has general consensus on:

1. Having one implementation of Arrow in Rust where we can focus would be better than 2 which split attention and resources
2. The technical underpinnings of arrow2 are more ergonomic

It is not clear to me if there is a consensus on:

1. How important the Apache Governance model is (please lend your opinions here!)
2. How important the stability of APIs / the specific versioning scheme (0.x vs 1.x or later)

Possible ideas for a way forward:

1. Switch datafusion to arrow2, making no changes to arrow-rs. It could be maintained by anyone who wished to contribute,
2. Bring arrow2 code into the arrow-rs repo, with appropriate IP clearance and adopt that as the officially maintained arrow implementation (*)
3. Start more actively porting the more ergonomic parts of arrow2 into arrow-rs to reduce the feature gap as suggested in Discussion: Switch DataFusion to using arrow2? datafusion#1532 (comment) by @tustvold
4. Others?

Option 2 leaves open the question of “how does arrow2 development move forward” – where would patches be sent, for example? I would hope we can find a way that is compatible with Apache governance, but I don't think we have a specific proposal yet, and it also depends in large part on what @jorgecarleitao is comfortable with

So, for any users of this crate not also in the DataFusion community, what are your hopes / needs / plans from this crate? How important is the apache governance to you? Please tell us your thoughts!

[andygrove]

My personal hope is that arrow2 can be donated to the ASF as a 0.x project. Given how widely used arrow2 is, I believe that it is becoming the de-facto implementation anyway and IMHO there would be benefits to having Apache governance.

For many of us who work for large corporations, we have to seek permission to contribute to open source projects. For example, I have permission to contribute to Apache Arrow and its subprojects, but I cannot contribute to arrow2 while it is an independent project (not that I would necessarily be contributing anyway, but others may be in a similar position).

I see no reason why the stable arrow crate cannot continue to exist and evolve independently of arrow2, assuming that there are contributors motivated to do so.

[alamb]

For the record, I would be willing to help maintain arrow2 if it were donated into the ASF doing things like coordinating release process / voting, bug fixing, answering questions, and the like

[houqp]

Thank you @alamb for starting and driving this discussion. Great summary on the current community consensus.

It is not clear to me if there is a consensus on:
How important the Apache Governance model is (please lend your opinions here!)

Personally, I think the Apache model works better for relatively slow moving monolith projects, while arrow2/parquet2 are fast evolving projects with a vision to be broken up into even smaller modular crates. @alamb has done an exceptional work on driving the arrow-rs releases. But seeing how much effort and time it takes, I would consider it a unnecessary overhead for arrow2 at its current stage. @jorgecarleitao was able to react to user feedbacks fast and release 3-4 new versions in a week for arrow2, this is simply not possible with the Apache Governance model. That said, I think the Apache voting process is very useful when you need high confidence on the quality of every single release and has a large diverse set of PMC members who can participate in the voting in a timely manner. But arrow2 seems still pretty far away from this.

@andygrove brought up a good point that it might become an issue for large corporations with restrictive open source contribution guide lines. This is the first time I am aware of this issue, previously I was under the impression that software license is all what matters. On the other hand, I am guessing ASF is not the only governance that's allowed? Perhaps we could help @jorgecarleitao come up with a different compatible governance model for arrow2 until it's ready for the ASF contribution? If Andy wants to contribute to arrow2 now but is blocked by lack of governance, then I would consider this a serious issue that we should address. Otherwise I would optimize for iteration velocity over governance until it becomes a real problem.

In short, from what I have seen so far, the upside from adopting the Apache governance model is to unblock potential contributions from big corporations. The downside is it will slow down our iteration process and potentially even disincentivize @jorgecarleitao from actively working on the project. Reading from his past emails, I get the feeling that he did try very hard to pass the IP clearance and donate arrow2 to ASF last year, but got frustrated by the bureaucracy. I am personally much more concerned about latter than the former.

How important the stability of APIs / the specific versioning scheme (0.x vs 1.x or later)

IMHO, this is not important as long as it is well communicated to the users. i.e. be explicit that we are special and please treat our 8.x as 0.x until we say otherwise. But Jorge has a strong opinion on this and want to strictly follow what the rest of the Rust ecosystem does. I also understand where he is coming from and respect his stance on this.

Switch datafusion to arrow2, making no changes to arrow-rs. It could be maintained by anyone who wished to contribute,

I agree with @andygrove on this. As long as there is community interests in this, we should probably still open arrow-rs up for contributions. This is not the result I want to see, but I have a feeling that this is likely what is going to happen :(

Start more actively porting the more ergonomic parts of arrow2 into arrow-rs

I think this is certainly doable, but then I stand by my previous comment that it won't be a good use of our time unless there is fundamental design tradeoffs in arrow-rs that are not compatible with arrow2's design. Simply replicating the design another project has is not a good reason to start a fork IMHO. I know @tustvold has a fairly strong opinion on this option and is more familiar with the parquet code base than I do, so perhaps he could help shed some light on this.

Option 2 leaves open the question of “how does arrow2 development move forward” – where would patches be sent, for example?

Just throwing out random idea here, one potential variant of option 2 is we use arrow-rs as the place to maintain stable arrow2 branches and let arrow2 iterate as fast as it could without the fear of introducing breaking changes. While the stable branch will cherry-pick compatible commits for a specific 0.x release that we want to maintain for X months. This way, we can still direct all contributions back to arrow2. The downside is I don't know how much interests the community has for a stable API considering we just decided to stop maintaining stable releases for arrow-rs.

[tustvold]

I know @tustvold has a fairly strong opinion on this option and is more familiar with the parquet code base than I do, so perhaps he could help shed some light on this.

I would just like to get away from this situation where we have two concurrent projects. It is just demoralizing, draining, and to be completely honest it just seems a tad unnecessary. Whilst I do not like the idea of porting stuff across, and yes it would be an annoying use of time, I am willing to contribute to such an effort if it sees an end to this situation. It overcomes what is otherwise a potentially indefinite political and bureaucratic discussion with pure technical brute force. It is very similar in my mind to option 2, simply changing the merge direction.

Ultimately I'm going to end up porting code regardless, I would prefer an outcome that allows me to save others the same effort 😄

[houqp]

I totally agree with you @tustvold . We are all engineers looking to solve interesting technical problems together, not playing political games after all.

[sunchao]

For many of us who work for large corporations, we have to seek permission to contribute to open source projects. For example, I have permission to contribute to Apache Arrow and its subprojects, but I cannot contribute to arrow2 while it is an independent project (not that I would necessarily be contributing anyway, but others may be in a similar position).

I'd like to echo @andygrove 's point here. The only reason we're able to contribute to Arrow/Parquet rust implementation is because it's under the governance of Apache. Otherwise, it'd be very hard for us.

[jorgecarleitao]

I would just like to get away from this situation where we have two concurrent projects. [...]

I agree. I agree that the situation is not productive. I am sorry that I caused frustration to people here.

Whilst I do not like the idea of porting stuff across, and yes it would be an annoying use of time, I am willing to contribute to such an effort if it sees an end to this situation.

I am also willing to contribute to such an effort.

What do you think about something to the effect of:

• Arrow2 is donated to Apache Arrow and its development ceases in jorgecarleitao/arrow2
• The core of arrow2 (array/, bitmap/, offsets.rs, types/) are lifted to a crate living in this repo (e.g. arrow-core or something).
• the core receives relevant methods from arrow-rs; add methods existing in arrow-rs with "deprecated" to give time for arrow-rs users to use them.
• arrow-rs' FFI of arrow2 is moved to a separate crate and replaces arrow-rs' one
• Arrow-rs' compute is migrated to use arrow-core
• Arrow-rs' IO except IPC is migrated to use arrow-core
• Arrow-rs' IPC IO is replaced by arrow2's implementation with necessary adjustments
• arrow-core will add RunEndArray (this is missing there atm)
• RecordBatch (arrow-rs) and Chunk (arrow2) co-exist to give room for both communities to use (in core or something else)
• Development and governance follows Apache and this repo of community-driven development.

This could result in the following changes to arrow-rs:

• ArrayData is removed
• It becomes interoperable with Vec but no longer aligned with cache lines.
• IPC support is improved with e.g. unsafe free, big endian support, mmap of IPC files
• There is code churn related to from vs from_slice (we can switch to arrow-rs names)
• slicing of arrays become easier (handling of offsets)

It would also end the arrow-arrow2 split e.g. removing the un-productive discussions around "which is better", and combine development efforts.

Some challenges:

• Arrow2 uses Box<dyn Array> as children to allow easy mutation; arrow-rs uses Arc<dyn Array>
• Arrow2 does not have TimestampArray nor DecimalArray, and instead sticks to the physical types only

[tustvold]

I am also willing to contribute to such an effort.

That would be fantastic, finding a way to unify our efforts would be amazing

What do you think about something to the effect of

Let me take some time to think about the implications of this and how we might do it in an incremental fashion. Whilst I have no particular affection for ArrayData, it is a fairly useful escape hatch and is important some more advanced use-cases (Vec's allocator support is still unstable IIRC).

[ritchie46]

I am also willing to contribute to such an effort.

I am also willing to help on this. And if this would go forward consider me as invested as long term maintainer of the project as well.

[alamb]

Thank you @jorgecarleitao @ritchie46 and @tustvold

Arrow2 is donated to Apache Arrow and its development ceases in jorgecarleitao/arrow2

I am willing to run the "Ip clearance" process to get the arrow2 codebase into the arrow-rs repository. I have done this before for object_store and while it takes some time I think it is worth it in this case.

As for the technical plans, the only thing I feel (very) strongly is that there is a migration path that doesn't involve a "all downstream crates need to rewrite the world" -- given the above discussions, it sounds like there are already some good thoughts on this matter so I trust it will be in good hands.

I would also be interested in what some of the other recently active contributors / maintainers of arrow-rs such @viirya @askoa and @iajoiner think of these ideas

FYI @liukun4515