feature: adapt predicate pushdown for mismatched nested/struct schemas

[adriangb]

As per discussion in #16461 (comment) we should be able to have datafusion/physical-expr/src/schema_rewriter.rs handle missing sub-fields in struct columns.

[alamb]

I think #16371 from @kosiew has the core piece of what is needed here -- namely the struct aware casting kernel

[adriangb]

@kosiew do you think we could incorporate the struct aware casting you added in #16371 into CastExpr and TryCastExpr?

[kosiew]

@adriangb,

incorporating the struct-aware casting logic from #16371 into CastExpr and TryCastExpr

Yes. I think it is a necessary step to explore the feasibility of rewriting Expr for schema adaptation.

A few thoughts on this:

1. Reusability of struct-aware casting kernels: The core struct-aware casting kernel introduced in Add nested struct casting support and integrate into SchemaAdapter #16371 was designed to recursively handle casting between nested struct types, dealing with missing or reordered fields. Integrating this logic into CastExpr and TryCastExpr would unify casting semantics across physical expressions, ensuring consistent behavior during predicate rewrite and evaluation.

2. Improved predicate pushdown: Predicate pushdown often requires projecting predicates down into struct subfields. If schemas differ between scans (e.g., fields missing or reordered), leveraging struct-aware casts would allow predicates to be adapted instead of dropped—thus generating more selective filters pushed down to data sources.

3. Implementation considerations: We would need to extend CastExpr and TryCastExpr to detect when the input/output types are structs and invoke the struct-aware casting logic appropriately. This means:

   • Adding recursive traversal logic in these expressions.
   • Handling missing subfields by either filling with NULLs or using default values as per the kernel's design.
   • Handling field reordering and type promotion gracefully.

4. Challenge: Rewriting the struct-aware casting logic from Add nested struct casting support and integrate into SchemaAdapter #16371 into manipulating expressions. Your expression rewrite prowess bends my mind.

[adriangb]

Question: will we be able to handle something like Dict(UInt32, List(List(Struct(...))))?

[alamb]

Implementation considerations: We would need to extend CastExpr and TryCastExpr to detect when the input/output types are structs and invoke the struct-aware casting logic appropriately. This means:

@kosiew I would actually suggest something different which is to add a function that "wraps" the arrow cast kernel

So it would look somethin glike

pub fn cast(
    array: &dyn Array,
    to_type: &DataType,
) -> Result<Arc<dyn Array>, ArrowError> {
  if matches!(to_type, DataType::Struct) {
    // call struct aware version in datafusion
   } else {
     // fallback to arrow implementatin
     arrow::compute::kernels::cast(array, to_type)
    }
}

Then you need to update all the places that directly call arrow cast with this new cast version, but the upside is that now everything would have the same (consistent) semantics

[adriangb]

I agree with you @alamb.

The issue I see with both approaches is going to be nested types: once you call the arrow cast kernel you can't take back control, so this approach won't work for List(Struct) and such. We'd essentially have to handle all non-leaf types in our wrapper and only delegate to arrow for the leaf types that can't possibly contain a struct / other type we want to handle.

[alamb]

I agree with you @alamb.

The issue I see with both approaches is going to be nested types: once you call the arrow cast kernel you can't take back control, so this approach won't work for List(Struct) and such. We'd essentially have to handle all non-leaf types in our wrapper and only delegate to arrow for the leaf types that can't possibly contain a struct / other type we want to handle.

I think delegating to arrow cast for leaf types makes sense, given its limitations casting nested types