[DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)

[alamb]

Is your feature request related to a problem or challenge?

I personally think making it easy to use DataFusion with the "open data lake" stack is very important over the next few months.

@julienledem wrote up a very nice piece describing The advent of the Open Data Lake

The high level idea is to make it really easy for people to build systems that query (quickly!) from parquet files stored on remote object store, including Apache Iceberg, Delta Lake, Hudi, etc.

You can already use DataFusion (and datafusion-cli) to query such data, but it takes non trivial effort to configure and tune for good performance. My idea is to make it easier to do so / make DataFusion better out of the box.

With that as a building block, people could/would build applications and systems targeting specific usecases

I don't yet fully understand where we currently stand on this goal, but I wanted to start hte discussio

Describe the solution you'd like

In my mind, the specific work this entails stuff like

• Making it easier to use iceberg/delta/hudi with DataFusion
• Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Make parquet reader in arrow-rs faster/better on remote object stores
• Making it eaiser to cache parquet metadata
• Support datafusion-cli access to public S3 buckets that do not require authentication #16299
• Improved experience when remote object store URL does not end in / #16302
• Support reading multiple parquet files via datafusion-cli #16303
• datafusion-cli: Use correct S3 region if it is not specified #16306

Describe alternatives you've considered

One specific item, brought up by @MrPowers would be to try DataFusion with the "10B row challenge" described in https://dataengineeringcentral.substack.com/p/10-billion-row-challenge-duckdb-vs .

I suspect it would be non ideal at first, but trying it to figure out what the challenges are would help us focus our efforts

Additional context

No response

[comphead]

What about remote HDFS files support? We have a contribution project https://github.com/datafusion-contrib/datafusion-objectstore-hdfs which supposed to query HDFS, but not sure how far we are with that

[alamb]

Yes I think HDFS would be another good target

Basically I want to make sure that it is as easy as possible to use DataFusion to query data that lives on remote systems (aka where the data is not on some local NVME but must be accessed over the network)

[jonathanc-n]

I forgot to mention this here, apache/iceberg-rust#700 (write support) is a really nice issue opened up in iceberg-rust. Getting the rust implementation of iceberg up and going would probably help out datafusion a bit on the data lake side of things.

[alamb]

I forgot to mention this here, apache/iceberg-rust#700 (write support) is a really nice issue opened up in iceberg-rust. Getting the rust implementation of iceberg up and going would probably help out datafusion a bit on the data lake side of things.

Yes, 100% -- one of my goals is to make it easy for this to "just work" with DataFusion. I think we are a bit away from it at the moment

[matthewmturner]

@alamb from a datafusion perspective which parts do you think are missing? I ask about just the datafusion perspective because i am assuming the owners of the relevant table formats will be implementing their spec / protocol.

[matthewmturner]

The way i was thinking about it the thing that makes it interesting / difficult is getting the semantics of each of the formats as part of datafusion which i would presume needs to be done as either SQL extensions and/or custom execution plans. Without knowing much about the specifics i thought that was already doable - but maybe im missing something.

[alamb]

@alamb from a datafusion perspective which parts do you think are missing? I ask about just the datafusion perspective because i am assuming the owners of the relevant table formats will be implementing their spec / protocol.

I am thinking mostly the list

• Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393
• Make an example showing how to use delta.rs / iceberg via FFI (and avoid version dependency hell)
• Make an example showing how to cache parquet metadata etc (not sure about this one yet -- I need to get it to the point where it is easy / convenient to read remote files first)

[matthewmturner]

Thanks @alamb

I plan to work on the second item - probably in December. I was really hoping to get a dft release out shortly where all the custom table providers (Delta / Iceberg / Hudi - and potentially Lance) were on DF v43 - but that might be wishful thinking.

Depending how the next week goes i will make a judgement call on whether or not to wait. If it isnt looking promising that everyone will be on v43 then my next priority will be implementing the FFI.

[blaginin]

I think one problem with the current implementation of external storages is that it's pretty hard to test properly. For example, the issue I solve in #13576 happened because right now, we only test that the external storage parameters are parsed, but we don’t even check if they’re parsed correctly.

Maybe we should start mocking aws/iceberg/... and move more towards integration testing? That way, we’d be more confident that our external storage support actually works 😅

[comphead]

@alamb I will try to cover how DataFusion works with remote HDFS files if that fits.
I'm planning to experiment with datafusion-objectstore-hdfs and make dev/tests/examples/docs so the user can easily query HDFS files from Datafusion CLI

[alamb]

Maybe we should start mocking aws/iceberg/... and move more towards integration testing? That way, we’d be more confident that our external storage support actually works 😅

Yes 100% -- maybe we can take a friendly look at the emultors used in object_store tests

https://github.com/apache/arrow-rs/blob/9047d99f6bf87582532ee6ed0acb3f2d5f889f11/.github/workflows/object_store.yml#L91-L184