sharing functionality between delta-*-rs / unitycatalog-rs and others

[roeap]

Motivations

Whenever we need to communicate with some sort of (cloud) service, authentication is usually a requirement. The means of authentication of course depend on the cloud / identity provider used for the API. Especially in cases like unity (or delta sharing for that matter) this often depends the cloud a service is running in. This usually means especially client libraries often either defer the question of acquiring auth tokens to the user, or depend on the various cloud SDKs to provide more convenience.

Back when we wrote object_store for the arrow ecosystem, we therefore went ahead and implemented the auth mechanics for various cloud providers and oAuth flows with minimal dependency footprint. This evolved into what I believe to be a production-grade http client tailored to cloud services. Since then I found myself regularly hoisting that code into different projects - like delta-sharing.

So we are trying to solve essentially the same problem in a lot of places ..

• Would you like to implement this feature? [y]

Solution

create a shared "cloud-client" crate that can be used to build higher level clients in various projects.

Alternatives

Not do this.

Additional context

[abrassel]

this sounds like a very fun project - I would love to be involved with the cloud-client crate. It seems particularly useful to delta-kernel-rs, another project I've been doing some work on.

[abhiaagarwal]

@roeap I actually had a very similar idea — I was planning on extending object_store's trait with a custom TemporaryCredentialProvider that would live as a subcrate of this project to handle the logic of dispensing temp credentials per cloud provider (as mentioned on #13). Maybe that could be folded into a larger cloud-client crate, there's obviously a demand for it!

[Jeadie]

@roeap I actually had a very similar idea — I was planning on extending object_store's trait with a custom TemporaryCredentialProvider that would live as a subcrate of this project to handle the logic of dispensing temp credentials per cloud provider (as mentioned on #13). Maybe that could be folded into a larger cloud-client crate, there's obviously a demand for it!

We're also interesting in a TemporaryCredentialProvider like paradigm in object-store. For us, we are needing to do some OAuth2.0 temporary credentials for some new ObjectStore types we're building.

[rtyler]

@roeap I agree with the solidity of the HTTP code in the object_store crate but would disagree with the credential handling logic 😆 I like the idea here, so long as it's a lot of async traits 😺

delta-rs was fundamentally broken on AWS for literal years 😱 recently because of the complexity of cloud-interactions we were not handling; fortunately the aws-sdk crates from AWS have simplified that. Have a look at the recently merged delta-io/delta-rs#2887 for some updates to our credentials.rs.

@abhiaagarwal that change also contains the AwsCredential implementation to glue temporary credentials into object_store. That sort of code is certainly worth bringing out.

From my perspective pursuing a cloud-client crate without dependencies on cloud's native crates (what object_store tries to do) is not likely going to be successful. I cannot speak for the Azure crates, but the AWS SDK crates are the standard interface, supported and maintained, by AWS for interaction with AWS.

What it sounds like you're wanting is something like jclouds for Rust, which is an ambitious project but definitely a useful one! Just please don't try to avoid cloud-specific dependencies like object_store does