Add SchemaAdapterFactory Support for ListingTable with Schema Evolution and Mapping #16583

kosiew · 2025-06-27T11:00:14Z

Which issue does this PR close?

This is the last of a series of PRs re-implementing #15295 to close #14757 by adding schema‐evolution support for:

listing‐based tables
with nested structs
in DataFusion.

Rationale for this change

This change introduces schema adapter support to improve DataFusion's ability to handle schema evolution scenarios in listing-based tables. It provides a more flexible and robust mechanism for adapting schemas between files and logical table definitions, including support for custom adapters that handle field renaming, type coercion, and column reordering.

What changes are included in this PR?

Introduced a new schema_adapter_factory field in ListingTableConfig and ListingTable
Added with_schema_adapter_factory and schema_adapter_factory() methods to both structs
Modified execution planning logic to apply schema adapters during scanning
Updated statistics collection to use mapped schemas
Implemented detailed documentation and example usage in doc comments
Added new unit and integration tests validating schema adapter behavior and error cases

Are these changes tested?

Yes, the PR includes a suite of tests to verify:

Default behavior without a custom schema adapter
Custom adapter impact on statistics mapping
Error propagation from schema mapping failures
Integration with listing scan execution

Are there any user-facing changes?

Yes:

Users can now provide a SchemaAdapterFactory in ListingTableConfig to customize how file schemas are interpreted relative to table schemas
Documentation is included to guide usage of schema evolution and custom mapping
Backwards-compatible; default behavior remains unchanged unless a factory is explicitly set

…sertions in `ListingTableConfig` methods

…ion support details

…ma_adapter method

…ableConfig

…chema state representation

…s_empty() for clarity

…l for clarity

adriangb

Leaving some questions / suggestions

datafusion/core/src/datasource/listing/table.rs

adriangb · 2025-07-02T19:45:23Z

datafusion/core/src/datasource/listing/table.rs

+/// # // Custom schema adapter for handling schema evolution
+/// # #[derive(Debug)]
+/// # struct EvolutionSchemaAdapterFactory;
+/// # impl SchemaAdapterFactory for EvolutionSchemaAdapterFactory {
+/// #     fn create(&self, projected_table_schema: SchemaRef, file_schema: SchemaRef) -> Box<dyn SchemaAdapter> {
+/// #         unimplemented!("Custom schema adapter implementation")
+/// #     }
+/// # }
+/// #


I don't think this should be here. It belongs alongside the SchemaAdapter trait / might already be there.

Before this PR, schema adapters was not used anywhere in the repo, except in tests.
Using schema adapter with a ListingTable is the first illustration of how to use schema adapters.
Therefore, added the example here.

Deleted this example, because there is an almost similar example appropriately located near fn with_schema_adapter_factory

I still think that (1) most users won't be using a SchemaAdapter and (2) linking to the existing docs on SchemaAdapter and maybe enhancing those should be enough.

adriangb · 2025-07-02T19:47:36Z

datafusion/core/src/datasource/listing/table.rs

+        // Note: We preserve existing options state, but downstream code may expect
+        // options to be set. Consider calling with_listing_options() or infer_options()
+        // before operations that require options to be present.
+        debug_assert!(
+            self.options.is_some() || cfg!(test),
+            "ListingTableConfig::with_schema called without options set. \
+             Consider calling with_listing_options() or infer_options() first to avoid panics in downstream code."
+        );
+
        Self {
-            table_paths: self.table_paths,
            file_schema: Some(schema),
-            options: self.options,
            schema_source: SchemaSource::Specified,
+            ..self


Is this related to this change or drive by?

This is a minor refactor because I added derive Default for ListingTableConfig

adriangb · 2025-07-02T19:47:59Z

datafusion/core/src/datasource/listing/table.rs

+    ///
+    /// # Example: Configuring Parquet Files with Custom Options
+    /// ```rust
+    /// # use std::sync::Arc;
+    /// # use datafusion::datasource::listing::{ListingTableConfig, ListingOptions, ListingTableUrl};
+    /// # use datafusion::datasource::file_format::parquet::ParquetFormat;
+    /// # let table_paths = ListingTableUrl::parse("file:///path/to/data").unwrap();
+    /// let options = ListingOptions::new(Arc::new(ParquetFormat::default()))
+    ///     .with_file_extension(".parquet")
+    ///     .with_collect_stat(true);
+    ///
+    /// let config = ListingTableConfig::new(table_paths)
+    ///     .with_listing_options(options);  // Configure file format and options
+    /// ```
    pub fn with_listing_options(self, listing_options: ListingOptions) -> Self {
+        // Note: This method properly sets options, but be aware that downstream
+        // methods like infer_schema() and try_new() require both schema and options
+        // to be set to function correctly.
+        debug_assert!(
+            !self.table_paths.is_empty() || cfg!(test),
+            "ListingTableConfig::with_listing_options called without table_paths set. \
+             Consider calling new() or new_with_multi_paths() first to establish table paths."
+        );
+
        Self {
-            table_paths: self.table_paths,
-            file_schema: self.file_schema,
            options: Some(listing_options),
-            schema_source: self.schema_source,
+            ..self
        }
    }


Same question as above

This is a minor refactor because I derive Default for ListingTableConfig

adriangb · 2025-07-02T19:48:18Z

datafusion/core/src/datasource/listing/table.rs

+    /// # #[derive(Debug)]
+    /// # struct MySchemaAdapterFactory;
+    /// # impl SchemaAdapterFactory for MySchemaAdapterFactory {
+    /// #     fn create(&self, _projected_table_schema: SchemaRef, _file_schema: SchemaRef) -> Box<dyn SchemaAdapter> {
+    /// #         unimplemented!()
+    /// #     }
+    /// # }


I think the example can just show setting the DefaultSchemaAdapterFactory

I retained this as an example of custom schema adapter factory....

I feel that showing an example using DefaultSchemaAdapterFactory here and linking to proper extensive docs on how to build a custom SchemaAdapter and the multiple layers of factories is more helpful.

adriangb · 2025-07-02T19:48:53Z

datafusion/core/src/datasource/listing/table.rs

+    /// # #[derive(Debug)]
+    /// # struct EvolutionAdapterFactory;
+    /// # impl SchemaAdapterFactory for EvolutionAdapterFactory {
+    /// #     fn create(&self, _projected_table_schema: SchemaRef, _file_schema: SchemaRef) -> Box<dyn SchemaAdapter> {
+    /// #         unimplemented!()
+    /// #     }
+    /// # }


Same as above, we can use DefaultSchemaAdapterFactory in the examples

... and changed this to DefaultSchemaAdapterFactory
with a note that they can refer to ListingTableConfig::with_schema_adapter_factory for custom example

adriangb · 2025-07-02T19:50:30Z

datafusion/core/src/datasource/listing/table.rs

+        let table_schema = self.schema();
+        match &self.schema_adapter_factory {
+            Some(factory) => {
+                factory.create(Arc::clone(&table_schema), Arc::clone(&table_schema))


Not this PR's fault but this is such a sad unfortunate API... why in the world do we pass in the same thing twice!?

Added create_with_projected_schema to the api

adriangb · 2025-07-02T19:51:44Z

datafusion/core/src/datasource/listing/table.rs

+        let schema_adapter = self.create_schema_adapter();
+        let (schema_mapper, _) = schema_adapter.map_schema(self.file_schema.as_ref())?;


Part of what smells wrong

I guess this is just for the column statistics?

datafusion/core/src/datasource/listing/table.rs

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

This reverts commit a300a30.

… ListingTableConfig

…method for convenience

…nto listing-config-14757

adriangb

Seems like a good improvement to me!

Btw, do you use ListingTable in production? There's plans to move it to an external repo or modularize it, it might be interesting for you to share your use case of it.

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

kosiew · 2025-07-04T06:32:23Z

@adriangb

Thanks for reviewing this.

Btw, do you use ListingTable in production? There's plans to move it to an external repo or modularize it, it might be interesting for you to share your use case of it.

No, I looked into it because of #14757

…on and Mapping (apache#16583) - Introduced a new `schema_adapter_factory` field in `ListingTableConfig` and `ListingTable` - Added `with_schema_adapter_factory` and `schema_adapter_factory()` methods to both structs - Modified execution planning logic to apply schema adapters during scanning - Updated statistics collection to use mapped schemas - Implemented detailed documentation and example usage in doc comments - Added new unit and integration tests validating schema adapter behavior and error cases

Add schema adapter factory and related code into ListingTableConfig

f779360

github-actions bot added the core Core DataFusion crate label Jun 27, 2025

kosiew added 6 commits June 27, 2025 19:32

refactor: rename None to Unset in SchemaSource and add debug as…

cf35911

…sertions in `ListingTableConfig` methods

docs: enhance documentation for ListingTableConfig with schema evolut…

fe07c22

…ion support details

docs: clarify schema adapter layering in create_file_source_with_sche…

b2dfb98

…ma_adapter method

test: add unit test for default schema adapter factory in ListingTable

1db3b88

docs: add examples for schema evolution and configuration in ListingT…

b1bf1c2

…ableConfig

refactor: rename None to Unset in SchemaSource for clarity in s…

86933eb

…chema state representation

kosiew changed the title ~~Add schema adapter factory and related code into ListingTableConfig~~ Add SchemaAdapterFactory Support for ListingTable with Schema Evolution and Mapping Jun 27, 2025

kosiew marked this pull request as ready for review June 27, 2025 13:04

kosiew added 4 commits June 28, 2025 10:11

fix: improve assertion for table_paths in ListingTableConfig to use i…

d28e615

…s_empty() for clarity

docs: update examples in ListingTableConfig to include ListingTableUr…

cb75d37

…l for clarity

Merge branch 'main' into listing-config-14757

217f781

Merge branch 'main' into listing-config-14757

17da443

kosiew mentioned this pull request Jul 2, 2025

Datafusion can't seem to cast evolving structs #14757

Closed

alamb requested a review from adriangb July 2, 2025 19:35

adriangb reviewed Jul 2, 2025

View reviewed changes

kosiew and others added 8 commits July 3, 2025 11:55

Update datafusion/core/src/datasource/listing/table.rs

5c719c2

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

Amend SchemaAdapterFactory documentation to be more specific

362767c

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

move custom schema adapter example to schema_adapter.rs

a300a30

Revert "move custom schema adapter example to schema_adapter.rs"

dd77340

This reverts commit a300a30.

docs: Remove redundant complete example for schema evolution setup in…

2c022b4

… ListingTableConfig

feat: Enhance SchemaAdapterFactory with create_with_projected_schema …

69ab53e

…method for convenience

Merge branch 'listing-config-14757' of github.com:kosiew/datafusion i…

614d369

…nto listing-config-14757

Merge branch 'main' into listing-config-14757

01fd2db

github-actions bot added the datasource Changes to the datasource crate label Jul 3, 2025

Merge branch 'main' into listing-config-14757

e23f87d

adriangb approved these changes Jul 4, 2025

View reviewed changes

Simplify comment for applying listing table schema adapter to source

cc86578

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

kosiew merged commit 8b9c1f1 into apache:main Jul 4, 2025
27 checks passed

		let schema_adapter = self.create_schema_adapter();
		let (schema_mapper, _) = schema_adapter.map_schema(self.file_schema.as_ref())?;

Add SchemaAdapterFactory Support for ListingTable with Schema Evolution and Mapping #16583

Add SchemaAdapterFactory Support for ListingTable with Schema Evolution and Mapping #16583

Uh oh!

Conversation

kosiew commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

kosiew commented Jul 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kosiew commented Jun 27, 2025 •

edited

Loading

kosiew Jul 3, 2025 •

edited

Loading