move projection handling into FileSource

[adriangb]

Part of #14993

This moves ownership of projections from FileScanConfig into FileSource.
Notably we do not do anything special with this in Parquet just yet: I leave it for a followup to actually use the projection expressions instead of column indices to e.g. generate the Parquet ProjectionMask directly from expressions (in particular to select leaves instead of roots for struct and variant access).

[AdamGS]

❤️

[adriangb]

This is the tracer bullet I've been using to guide this change:

use arrow::util::pretty::pretty_format_batches;
use arrow_schema::DataType;
use datafusion::{
    common::Result,
    prelude::{ParquetReadOptions, SessionContext},
};

#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();
    let df = ctx
        .read_parquet(
            "data/",
            ParquetReadOptions::default()
                .table_partition_cols(vec![("b".to_string(), DataType::UInt32)]),
        )
        .await?;
    let res = df
        .select_exprs(&["b", "a", "a = b", "a * 2"])?
        .collect()
        .await?;

    println!("Result:\n{}", pretty_format_batches(&res)?);

    Ok(())
}

I plan to do a review and cleanup, handle the protobuf stuff (I know that needs to be changed) and then fix all the rest of the tests. It looks like we have 36 failing tests, not too bad, and a lot of them look related.

[adriangb]

Down to 4 failing tests! All proto related.

[adriangb]

TODO: do we need to dynamically generate a "full" projection that includes table partition columns? Or should that be the default that each FileSource initializes itself with (then we don't even need to check here!)?

[adriangb]

I'm marking this as ready for review. I expect more changes / cleanup is needed, I already have some ideas, but I'd like some initial feedback. cc @AdamGS @XiangpengHao @waynexia since you've all expressed interest.

[adriangb]

To make life easier for reviewers I've opened a couple smaller hopefully "no brainer" PRs to break work out of this one:

• Consolidate ArrowFileSource and ArrowStreamFileSource #18720
• Move statistics handling into FileScanConfig #18721
• Misc improvements to ProjectionExprs #18719