Consider adding Polars as possible data processing framework.

[ritchie46]

I see you already support DataFusion and Rust DataFrame as well, which both use apache arrow. It would be nice if we could read/write from/to Polars as well.

If you'd be willing to, I could initialize a PR that sets that up.

[houqp]

Yes, please do, we will be more than happy to merge the PR. Make sure you update the readme to mention Polars as well :)

[ritchie46]

Sorry for the delay, I will work on is. Is it ok to also add an implementation based on arrow2 (under a feature flag)?

[houqp]

@ritchie46 yes, we want to move to arrow2 in the long run as well :) but it would be good if we can manage it with a feature flag because it's going to take some amount of time before we can migrate kafka-delta-ingest to arrow2.

[houqp]

If arrow2 feature flag is not possible, we can also maintain an official arrow2 branch

[ritchie46]

@ritchie46 yes, we want to move to arrow2 in the long run as well :)

Great! Polars already runs great on arrow2, and this way, I hope to improve third party support as well. :)

If arrow2 feature flag is not possible, we can also maintain an official arrow2 branch

Yes, I don't know how deep embedded arrow is in delta-rs?

[houqp]

Yes, I don't know how deep embedded arrow is in delta-rs?

It's not deep, I think the migration should be pretty straight forward. The hard part is isolating it into a feature flag so we can support both arrow and arrow2 in the short term.

[wseaton]

Would love to see this (specifically an arrow2 feature flag), could consolidate a lot of my custom code around parquet partitioning and writing to object storage + gain the benefits of delta lake on top of that.

[ritchie46]

I got bounced on the arrow2 migration. It was not so trivial to feature gate indeed. So that would probably need a separate branch?

[houqp]

@ritchie46 I actually got something mostly working 2 weeks ago, but got distracted with datafusion and roapi so wasn't able to complete the PoC. Let me try to get something pushed out this weekend so we can take a look at it together to see if that's the right path forward.

[houqp]

@ritchie46 I sent my PoC to #465.

[ritchie46]

@ritchie46 I sent my PoC to #465.

Wow, you were even able to feature gate it. kudos! 🙌

[mohitreddy1996]

@ritchie46 and @houqp thanks a lot for pushing the PoC. I was wondering if this is feature complete and any example I could refer to get started?

[roeap]

@mohitreddy1996 - if it is about reading delta tables with polars, the methods read_delta and scan_delta were recently added.

Beyond that we are also looking into making internal data-processing more flexible in terms of the processing engine being used. To make this work with polars the arrow2/parquet2 creates need to be used internally, which is possible today. But fully integrating polars internally still requires some major work and we unfortunately cannot give a concrete date if and when that will be supported.