Field names containing periods such as f.c cannot work

[hengfeiyang]

Describe the bug
Field names containing periods such as f.c cannot work.

To Reproduce
Steps to reproduce the behavior:

use std::sync::Arc;

use datafusion::arrow::array::Int32Array;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::datasource::MemTable;
use datafusion::error::Result;
use datafusion::from_slice::FromSlice;
use datafusion::prelude::{col, lit, SessionContext, Expr};
use datafusion_common::Column;

/// This example demonstrates how to use the DataFrame API against in-memory data.
#[tokio::main]
async fn main() -> Result<()> {
    // define a schema.
    let schema = Arc::new(Schema::new(vec![Field::new("f.c", DataType::Int32, false)]));

    // define data.
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(Int32Array::from_slice(&[1, 10, 10, 100]))],
    )?;

    // declare a new context. In spark API, this corresponds to a new spark SQLsession
    let ctx = SessionContext::new();

    // declare a table in memory. In spark API, this corresponds to createDataFrame(...).
    let provider = MemTable::try_new(schema.clone(), vec![vec![batch]])?;
    ctx.register_table("t", Arc::new(provider))?;
    let df = ctx.table("t")?;
    let df = df.with_column("f2", lit("hello"))?;

    // construct an expression corresponding to "SELECT * FROM t WHERE f.c = 10" in SQL
    let filter = Expr::Column(Column {
        relation: None,
        name: "f.c".into()
      }).eq(lit(10));

    let df = df.filter(filter)?;

    // print the results
    df.show().await?;

    Ok(())
}

Run the example will got error:

Error: SchemaError(FieldNotFound { qualifier: Some("f"), name: "c", valid_fields: Some(["t.f.c"]) })

Expected behavior
expect it can work.

Additional context
DataFusion 12.0

[alamb]

Does this work if you run a query (note the double quotes " around f.c)?

SELECT * FROM t WHERE "f.c" = 10

[hengfeiyang]

@alamb In SQL layer it can work, but in DataFrame it can't work.

[alamb]

@hengfeiyang I think you may have to create an explicit column reference -- for example instead of col("f.c") perhaps you could do something like

Expr::Column(Column {
  relation: None,
  name: "f.c".into()
})

[hengfeiyang]

@alamb Thanks, the filter can work now. But my real work is with_column, I updated my example code above, added a line:

    let df = df.with_column("f2", lit("hello"))?;

You can have a try, this line will cause the error too, Do you have any idea? thank you.

[alamb]

You can have a try, this line will cause the error too, Do you have any idea? thank you.

I think it is a bug in with_column (basically it is using col as well). I will create a PR to fix it. Thank you for the report @hengfeiyang

[alamb]

#3700