Can't serialize example ExecutionPlan to substrait
#9299
Labels
documentation
Improvements or additions to documentation
enhancement
New feature or request
substrait
Comments
alamb
added
enhancement
|
I am interested in taking a look at this. I was able to reproduce the error using the example above: Error: Substrait("Unsupported plan in Substrait physical plan producer: MemoryExec: partitions=1, partition_sizes=[1]\n") |
|
I looked into the example and the physical plan substrait producer/consumer code. Unfortunately for physical plans, the subtrait consumer and producer are only implemented for Here is an example which makes it further than the above but panics on the roundtrip assertion: use datafusion::prelude::*;
use std::collections::HashMap;
use datafusion::error::Result;
use datafusion_substrait::physical_plan;
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()>{
// Create a plan that scans table 't'
let ctx = SessionContext::new();
let testdata = datafusion::test_util::parquet_test_data();
ctx.register_parquet(
"alltypes_plain",
&format!("{testdata}/alltypes_plain.parquet"),
ParquetReadOptions::default(),
)
.await?;
let df = ctx
.sql(
"SELECT * from alltypes_plain",
)
.await?;
let physical_plan = df.create_physical_plan().await?;
// Convert the plan into a substrait (protobuf) Rel
let mut extension_info= (vec![], HashMap::new());
let substrait_plan = physical_plan::producer::to_substrait_rel(physical_plan.as_ref(), &mut extension_info)?;
// Decode bytes from somewhere (over network, etc.) back to ExecutionPlan
let physical_round_trip = physical_plan::consumer::from_substrait_rel(
&ctx, &substrait_plan, &HashMap::new()
).await?;
assert_eq!(format!("{:?}", physical_plan), format!("{:?}", physical_round_trip));
Ok(())
}And here is the panic output: You can see that the round trip lost many details about the I think if we want to include a user facing example of a physical plan substrait roundtrip, we will need to cut a ticket to complete the implementation of It looks like #5176 built the initial framework for serializing physical plans, but it hasn't been picked up since then. |
This was referenced Feb 26, 2024
I filed #9347 to track this Thank you for looking into this @devinjdangelo I also made #5173 an epic and added this ticket and #9347 to it |
|
I added to the substrait support epic: #5173 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge?
While working on an example for serializing substrait plans (see #9260 PR), I found I could not write an example for serializing an execution plan. The feature is still not complete enough
Describe the solution you'd like
What I would like:
I would like to add the example (or something like it) to
datafusion/substrait/src/lib.rsand haveit work.
//! # Example: Serializing [`ExecutionPlan`]s //! //! This functionality is still under development and only works for a small subset of plans //! //! ``` //! # use datafusion::prelude::*; //! # use std::collections::HashMap; //! # use std::sync::Arc; //! # use datafusion::error::Result; //! # use datafusion::arrow::array::Int32Array; //! # use datafusion::arrow::record_batch::RecordBatch; //! # use datafusion_substrait::physical_plan; //! # #[tokio::main(flavor = "current_thread")] //! # async fn main() -> Result<()>{ //! // Create a plan that scans table 't' //! let ctx = SessionContext::new(); //! let batch = RecordBatch::try_from_iter(vec![("x", Arc::new(Int32Array::from(vec![42])) as _)])?; //! ctx.register_batch("t", batch)?; //! let df = ctx.sql("SELECT x from t").await?; //! let physical_plan = df.create_physical_plan().await?; //! //! // Convert the plan into a substrait (protobuf) Rel //! let mut extension_info= (vec![], HashMap::new()); //! let substrait_plan = physical_plan::producer::to_substrait_rel(physical_plan.as_ref(), &mut extension_info)?; //! //! // Decode bytes from somewhere (over network, etc.) back to ExecutionPlan //! let physical_round_trip = physical_plan::consumer::from_substrait_rel( //! &ctx, &substrait_plan, &HashMap::new() //! ).await?; //! assert_eq!(format!("{:?}", physical_plan), format!("{:?}", physical_round_trip)); //! # Ok(()) //! # }When you run this test today you get an error about "mem provider not implemented" or something like that
Describe alternatives you've considered
No response
Additional context
I think making this work would be a matter of implementing serialization of MemTable / MemExec perhaps
The text was updated successfully, but these errors were encountered: