S3 Support

[ehenry2]

Now that there is a PR in-flight for remote storage systems (#811 ), I wanted to ask what the plan was to implement s3 support. Is this something that will live within the datafusion project or maintained outside? Also, does anyone plan on working on this, or is this something that you are looking for a contributor for?

Also, I'm new to the project, so what is the best way to ask these kind of questions...github issues or on the email list? thanks!

[Dandandan]

My personal feeling is that we probably don't want to have every storage system inside the DataFusion crate by default but either:

• Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions
• Have some optional features (e.g. s3) living inside the crate.

Wydt @alamb @houqp @andygrove @jorgecarleitao

[jimexist]

My personal feeling is that we probably don't want to have every storage system inside the DataFusion crate by default but either:

• Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions

• Have some optional features (e.g. s3) living inside the crate.

Wydt @alamb @houqp @andygrove @jorgecarleitao

A separate crate is a good idea.

How would you think of keeping the code as part of the workspace?

[ehenry2]

Another question I wanted to throw out too, on implementation details (probably putting the cart before the horse, but wanted to ask anyway), was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this.

[alamb]

Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions

Yes I think separate crates is a very good idea to avoid datafusion requiring a massive dependency stack and to keep compile time reasonable.

How would you think of keeping the code as part of the workspace?

I think this is a reasonable for "core" integrations, though going this path also subjects those crates to the apache arrow governance model (among other things slower release cycle) which may not be needed here

Maybe we could start it as its own repo / separate crate with whoever does it and then we can figure out if we want to bring it back into apache arrow if any.

was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this.

FWIW we took this approach in IOx (starting with Rusto) and it has been working well for us: https://github.com/influxdata/influxdb_iox/blob/main/object_store/Cargo.toml#L21-L23 (we have been using it in AWS for several months now and we have not hit any issues with it)

[houqp]

I also think separate crate is a good idea. The design that @yjshen came up with made it very easy to implement different remote storage backends as self-contained plugins.

Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at apache/datafusion-sqlparser-rs#308 (comment)? We could use it as the incubator for various experiments that need collaborations instead of using personal namespace.

How would you think of keeping the code as part of the workspace?

I think the upside of this is we won't need to deal with IP clearance if we ever need to donate it back to datafusion. The downside is what Andrew said earlier, slower release cycle due to apache governance overhead, which is not too bad for a stable project. For newer and fast evolving projects, it might slow things down unnecessarily.

[yjshen]

Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at apache/datafusion-sqlparser-rs#308 (comment)? We could use it as the incubator for various experiments that need collaborations instead of using a personal namespace.

Wow, that's great!

[yjshen]

If someone may be interested, I will also work on HDFS support right after this.

[andygrove]

Maybe it's finally the time to create that general Rust database Github Org that @andygrove mentioned at sqlparser-rs/sqlparser-rs#308 (comment)? We could use it as the incubator for various experiments that need collaborations instead of using personal namespace.

If we are developing crates that are extensions to DataFusion then I would be nervous about creating a separate org & community with its own governance for these crates. I think it would be fine for generic database-related crates that don't have "arrow" or "datafusion" in the name, such as the sqlparser-rs crate.

We already have the ability to create arrow-experimental-* repos where we do not need so much governance. Perhaps we should try that route first?

[ehenry2]

Wanted to reopen the discussion here...do we have a consensus on which approach we want to go with? Now that the first set of remote storage PRs are merged, this can now be implemented.

[alamb]

I think there is a consensus on the architecture within datafusion, now that #950 has been merged

I think we now would need to rework the various table providers (CsvExec, ParquetExec, etc) to use that interface rather than File or Read directly, and then we would be ready to create an S3 plugin for DataFusion

[ehenry2]

@alamb That makes sense, my thought though is that s3 support can be worked on in parallel (although not able to utilized until the table providers are updated) as the scaffolding is there with the Object Reader and Object Store traits. In order to do that, however, either a repo would need to be created (or one of the other approaches given previously adopted).

[houqp]

Taking @andygrove 's comment into account, unless anyone volunteers to help create the arrow-experimental-datafusion-s3 repo, it might be easier to follow @jimexist 's suggestion to create and manage the s3 plugin crate inside datafusion's workspace. Downstream applications can pull in the plugin dependency by git hash if we are behind on official releases. Also I believe we can release alpha/beta versions to crates.io without requiring an official vote?

[alamb]

Starting to work on an experimental crate within arrow-datafusion seems like a good approach to me.

I haven't tracked / studied the requirements for releasing alpha/betas in fine detail, but I think as long as they are clearly marked as not official apache releases, it would be fine to put them on crates.io