Is ObjectStoreSchemaProvider Still Needed?

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The ObjectStoreSchemaProvider is a SchemaProvider that provides the ability to register ObjectStore and ListingTable. Effectively it is a MemorySchemaProvider but which maintains its own ObjectStoreRegistry instead of using the one on RuntimeEnv.

I'm quite possibly missing something, but I'm not entirely sure why this is needed?

Describe the solution you'd like

If it isn't needed, it would be good to remove it.

Describe alternatives you've considered

If it is needed, we should keep it 😄

Thoughts @matthewmturner ?

[matthewmturner]

FYI #1836 and #1863 is background conversation on the API design for this.

The original idea was to use this as shown below (slightly updated from how it was discussed in the above issue) - i.e. easily register multiple tables from a prefix into a schema.

let object_store = S3FileSystem::default();
let schema = ObjectStoreSchemaProvider::new();
schema.register_store(object_store);
let tables = object_store.list_dir("s3://active/schema1");
tables.iter().map(|file|  {
    let config = ListingTableConfig::new(object_store, file).infer().await?;
    let name = extract_name_from_file(&file);
    schema.register_listing_table(name, file, config);
}

Thinking on it now, i think the same could be done without the need for this abstraction, like the following:

let object_store = S3FileSystem::default();
let schema = MemorySchemaProvider::new();
let tables = object_store.list_dir("s3://active/schema1");
tables.iter().map(|file|  {
    let config = ListingTableConfig::new(object_store, file).infer().await?;
    let table = ListingTable::try_new(config)?;
    let name = extract_name_from_file(&file);
    schema.register_table(name, table)?;
}

If you agree, then yes I do think we can remove ObjectStoreSchemaProvider.

[tustvold]

Created #2665