enhancement (datafusion-cli): Add support for glob patterns in CREATE EXTERNAL TABLE commands #16387
+337
−5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partly closes #16303
The purpose of this PR is to enable using CREATE command with glob pattern and a URL scheme - i.e.,
Its currently possible to create an external table using this syntax just for local files:
Therefore, the purpose here is to enable support for glob support also for remote url scheme.
The implementation involves some sort of workaround - it intercepts
create_plan()
, and when the table involves a glob pattern and remote scheme then it creates it as a ListingTable. Part of the reason for this approach is the fact that DataFusion core modules use ListingTable::parse() method in its core modules, which only takes a glob pattern when it invovles local files (see/datafusion/core/src/datasource/listing_table_factory.rs
for example).