Skip to content

Automatically check "invariants" #13652

Closed
@alamb

Description

@alamb

Is your feature request related to a problem or challenge?

I extracted this from #13651 so it was more visible

During upgrade, downstream systems often experience issues due to implicit changes (not explicit API changes) of LogicalPlans that DataFusion code begins relying on, and which result in unintended consequences when upgrading to a new version of DataFusion (see #13525).

Describe the solution you'd like

The idea is to make the current implicit assumptions ("Invariants" in more formal language)( explict and automatically check them.

Examples of implicit assumptions:

  1. Schema column names can't be repeated (this is explicitly mentioned on [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525)
  2. Inputs to UnionExec must have the same schema
  3. ...

Describe alternatives you've considered

I like the approach @wiedld took in #13651 :

  • define the invariants
  • check the invariants for extensible interfaces (which may be user defined)
  • throw the error closer to the problem (rather than weird behavior later)

Additional context

Sub tasks:

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions