Skip to content

Make the conversion from logical to physical plan async #1012

@rdettai

Description

@rdettai

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As discussed in #1010, it would add a lot of flexibility to the planning to make it possible to use async methods. Thanks to #962, the statistics resolution can be delayed to the creation of the physical plan. Adding async to TableProvider.scan(), the method that converts the logical plan to the physical plan, will make it possible to have async statistics resolution.

Describe the solution you'd like
Make the trait method TableProvider.scan() async and propagate that change to the outer APIs (mostly ExecutionContext.create_physical_plan(&LogicalPlan))

Describe alternatives you've considered

  • also add statistics to optimizers: that could be a next step but it is not required for now
  • leave the planning sync and force TableProvider implementers to make their async methods sync: syncifying in Rust is pretty hard (cloud-reader-rs is an example of this) and it is a huge constraint for implementers

Additional context
This will also help the useage of the ObjectStore abstraction added in #811.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions