Is there a way to go from `ObjectStore` to `(URL, opts)`?

[criccomini]

Which part is this question about

ObjectStore's parse_url.

Describe your question

We're looking at converting between ObjectStores and URLs in SlateDB:

slatedb/slatedb#533 (comment)

URL to ObjectStore seems pretty straight forward with parse_url, but it's unclear how we might go from ObjectStore to URL. Is there some get_url() method we can call on the ObjectStore trait? I poked around but didn't find anything. I suspect I'm missing something obvious.

Additional context

[tustvold]

This is not currently supported, you will need to store the URL alongside the ObjectStore in whatever you are using to cache them.

See also #259

[criccomini]

Gotcha. Would you be open to a PR to generate/expose the URL via a get_url method in the ObjectStore trait? The reason I ask is that SlateDB takes ObjectStores in its constructor. Users are constructing the ObjectStore outside of our code. We could, of course, ask them to supply a url and opts, and call parse_url while caching the URL. But that seems less desirable since it makes it harder to do dependency injection for testing. It also feels a bit clunky.

[tustvold]

See #259 (comment)

TLDR I am not sure how this would actually work.

Edit: I'd suggest looking at DataFusion's ObjectStoreRegistry abstraction, to see how it handles this. There have been some discussions about upstreaming this, but they've not progressed further.

[criccomini]

Oh, thanks for the pointer. ObjectStoreRegistry looks exactly like what we'd want. Upstreaming it would be really nice. Would you be open to a PR? (I don't want to deal with updating the data fusion code, though 😅 )

Regarding the comment in #259, I'm playing around with the code a bit just to get a feel for it.

[tustvold]

If someone wanted to upstream it, I think that would be valuable

[alamb]

Oh, thanks for the pointer. ObjectStoreRegistry looks exactly like what we'd want. Upstreaming it would be really nice. Would you be open to a PR? (I don't want to deal with updating the data fusion code, though 😅 )

Regarding the comment in #259, I'm playing around with the code a bit just to get a feel for it.

Just adding the ObjectStoreRegistry to this create is enough -- we can find relevant DataFusion community members to update DataFusion

Thank you for your help @criccomini

[alamb]

For anyone else curious, here is a link to the docs for ObjectStoreRegistry

[criccomini]

PR here: #348