Cannot insert with DMLStatement into a table with non-nullable fields #8763
Description
Describe the bug
When a table is created with a non-nullable Field, executing a DMLStatement logical plan to write to that table always fails with the error Column [name] is declared as non-nullable but contains null values. This happens no matter what values I try to insert.
To Reproduce
The following snippet creates a table called t1 with a non-nullable int32 Field called a. When we try to add an integer into this table, it errors.
use async_std::task;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::common::{DFSchema, ScalarValue};
use datafusion::datasource::MemTable;
use datafusion::logical_expr::{DmlStatement, Expr, LogicalPlan, Values};
use datafusion::logical_expr::WriteOp::InsertInto;
use datafusion::prelude::SessionContext;
use std::sync::Arc;
fn main() {
let table_name = "t1";
let field = Field::new("a", DataType::Int32, false);
let ctx = SessionContext::new();
let schema = Schema::new(vec![field]);
let df_schema = DFSchema::try_from(schema.clone()).unwrap();
match MemTable::try_new(schema.clone().into(), vec![Vec::<RecordBatch>::new()]).ok() {
Some(mem_table) => {
let _ = ctx.register_table(table_name.clone(), Arc::new(mem_table));
}
None => panic!("An unexpected error occured creating the table"),
};
let values = vec![vec![Expr::Literal(ScalarValue::Int32(Some(1)))]];
let values_plan = LogicalPlan::Values(Values {
schema: df_schema.clone().into(),
values: values.clone(),
});
let dml_plan = LogicalPlan::Dml(DmlStatement {
table_name: table_name.into(),
table_schema: df_schema.clone().into(),
op: InsertInto,
input: values_plan.into(),
});
// This step always errors out
// Error:
// ArrowError(InvalidArgumentError("Column 'a' is declared as non-nullable but
// contains null values"))'
let dataframe = task::block_on(ctx.execute_logical_plan(dml_plan.clone()))
.expect("Could not execute plan");
let _ = task::block_on(dataframe.collect()).expect("Could not collect dataframe");
println!("Success!");
}Expected behavior
We would expect this insert to succeed, as the values we are inserting are not null.
Additional context
No response