Add CastColumnExpr for struct-aware column casting

[kosiew]

Which issue does this PR close?

• Closes Add a dedicated physical expression for struct-aware casting that mirrors the existing cast_column helper semantics. #17760

Rationale for this change

The planner sometimes needs to rewrite the physical representation of a resolved column to a new schema/field while preserving nested-structure semantics. The existing CastExpr doesn't carry the input/target field metadata required to perform struct-aware casts (correct nested field ordering, nullability, and null-padding for missing children). Adding a dedicated CastColumnExpr lets the execution layer call into datafusion_common::nested_struct::cast_column and guarantees casts behave correctly for both array and scalar values.

This change is focused on execution-time casting semantics (schema-aware casts) and does not attempt to modify planner/optimizer behaviour.

What changes are included in this PR?

• Add new physical expression implementation: datafusion/physical-expr/src/expressions/cast_column.rs.

  • Defines CastColumnExpr struct which contains:

    • expr: Arc<dyn PhysicalExpr> — child expression producing the value to be cast.
    • input_field: FieldRef — resolved input field metadata.
    • target_field: FieldRef — desired output field metadata.
    • cast_options: CastOptions<'static> — forwarded to cast_column.

  • Implements PhysicalExpr for CastColumnExpr:

    • data_type and nullable reflect the target_field.
    • evaluate handles both ColumnarValue::Array and ColumnarValue::Scalar by delegating to datafusion_common::nested_struct::cast_column, then converting results back to ColumnarValue.
    • children, with_new_children, fmt_sql, return_field implemented for planner/execution compatibility.

  • Manual PartialEq and Hash impls to accommodate the Arc<dyn PhysicalExpr> child.

• Export the new expression in datafusion/physical-expr/src/expressions/mod.rs (mod cast_column; and pub use cast_column::CastColumnExpr;).

• Add unit tests in cast_column.rs covering:

  • Primitive array casts (Int32 -> Int64).
  • Struct array where the target struct has a missing child (null-padding behavior).
  • Nested struct casts (preserve nested layout, add missing nested children with nulls).
  • Struct scalar casts (scalar -> casted struct scalar).

• Add module-level docs/comments describing intent and usage.

Are these changes tested?

Yes — unit tests included in cast_column.rs exercise array & scalar cases, nested structs, missing children, null-padding and simple primitive casts. Tests added:

• cast_primitive_array — cast Int32Array to Int64Array.
• cast_struct_array_missing_child — source struct has fields [a, b], target struct requests [a, c] and expects c to be all-null.
• cast_nested_struct_array — nested struct casting where inner struct adds a new child field that must be null-padded.
• cast_struct_scalar — casting a struct literal (scalar) and preserving result as a ScalarValue::Struct with casted children.

All tests are Rust unit tests using RecordBatch and Column/Literal helpers already available in the crate.

Are there any user-facing changes?

• Public API: CastColumnExpr is exported from datafusion::physical_expr::expressions and can be constructed/used by callers who build physical expression trees. This is primarily intended for internal planner/code that needs to perform schema-aware casting of resolved columns.

• No changes to existing SQL surface or planner rules are included in this PR — it only adds a building block for use by other components (e.g. schema rewriters or planner nodes).

• No behaviour change for existing code that doesn't use CastColumnExpr.

[comphead]

Thanks @kosiew WDYT would be that possible to have e2e slt tests?

[kosiew]

@comphead

would be that possible to have e2e slt tests?

Thanks for the suggestion! At the moment this PR just introduces the CastColumnExpr building block plus its focused unit coverage, without wiring it into the planner or schema-adapter path that SQLLogicTest exercises.
It's the first sub-issue of #17330
Until that follow-up lands, there isn’t a SQL surface that would actually emit this node, so an SLT case would stay unreachable and wouldn’t add new signal.

Once we hook the adapter/planner up to construct CastColumnExpr, I agree that an end-to-end SLT is the right next step and I’ll plan to cover it in that integration change.

[comphead]

thanks @kosiew

[kosiew]

Thanks @comphead for your review.

[alamb]

Thank you for this PR @kosiew

[alamb]

What is your long term vision of CastExpr and CastColumnExpr?

It seems like CastExpr can be totally replaced with CastColumnExpr as CastColumnExpr is more general.

datafusion/datafusion/physical-expr/src/expressions/cast.rs

Lines 44 to 53 in 631f9ab

	/// CAST expression casts an expression to a specific data type and returns a runtime error on invalid cast
	#[derive(Debug, Clone, Eq)]
	pub struct CastExpr {
	/// The expression to cast
	pub expr: Arc<dyn PhysicalExpr>,
	/// The data type to cast to
	cast_type: DataType,
	/// Cast options
	cast_options: CastOptions<'static>,
	}

However, it seems there are a few places in the code (in the core repo and elsewhere) that look for CastExpr directly: https://github.com/search?q=repo%3Aapache%2Fdatafusion%20downcast_ref%3A%3A%3CCastExpr%3E&type=code

In other words, do you think it would be better to update the existing CastExpr to handle Fields rather than introduce a new CastColumnExpr ?

[kosiew]

The initial plan was to
Replace the fallback CastExpr creation with CastColumnExpr instantiation when rewriting columns. when I work on #17765
In view of your findings above, I will change #17765 to instead
update the existing CastExpr to handle Fields in place of CastColumnExpr

[alamb]

Sounds like a good plan, thank you

[alamb]

BTW thank you for pushing this along. I actually think moving fields down through more of DataFusion will help many things, including logical types / Arrow Extension Type support , as described in
Implementing User Defined Types and Custom Metadata in DataFusion