`push_down_projection` optimization fails when using variables

[rgwood]

Describe the bug

Hello and thank you for writing+maintaining DataFusion!

When upgrading from DataFusion 18, we noticed an unexpected warning that started in #4465 by @jackwener (which shipped in DataFusion 20):

WARN datafusion_optimizer::optimizer: Skipping optimizer rule 'push_down_projection' due to unexpected error: Error during planning: required columns can't push down

To Reproduce

I've put together a minimal Rust repro here: https://github.com/rgwood/df-repro

In that repro, I use a VarProvider to define an integer variable @var.

I then use that variable in the WHERE clause of a SQL query: SELECT foo FROM csv_table WHERE bar > @var

When evaluating the query, a warning indicates that push_down_projection was skipped:

WARN datafusion_optimizer::optimizer: Skipping optimizer rule 'push_down_projection' due to unexpected error: Error during planning: required columns can't push down, columns: {Column { relation: Some(Bare { table: "csv_table" }), name: "bar" }, Column { relation: None, name: "@var" }, Column { relation: Some(Bare { table: "csv_table" }), name: "foo" }}

Expected behavior

I expect the push_down_projection optimization to succeed without a warning.

Additional context

I'm relatively new to DataFusion, but it seems like push_down_projection is incorrectly interpreting the variable @var as a column when it should not be.

[alamb]

Thanks for the report @rgwood -- @jackwener does this sound familiar to you?

[alamb]

@rgwood would it be possible to provide a reproducer? I tried briefly but it seems like we don't have any way to set these variables via SQL yet

[rgwood]

would it be possible to provide a reproducer?

Sure, I’ve put together a minimal reproduction here: https://github.com/rgwood/df-repro

[jackwener]

Thank you @rgwood I will take a look.

[rgwood]

Wow, that was a super quick response+fix. Thank you @jackwener and @alamb!

[alamb]

All the kudos to @jackwener 🙏