Provide documentation of expose APIs to enable handling of type coercion at UNION plan construction.#12142
Merged
comphead merged 3 commits intoapache:mainfrom Aug 26, 2024
Conversation
…ly pub interfaces
…nvolved in type coercion
…ess type coercion
github-actions
bot
added
logical-expr
Contributor
Author
|
Note to @alamb -- exposing the rewriter api (as suggested here), enables the union type-coercion with less of the schema shenangians and overhead (vs the full analyzer). Gives the user options. |
comphead
reviewed
Aug 24, 2024
comphead
reviewed
Aug 24, 2024
| /// or alternatively, merge the union input schema using [`coerce_union_schema`] and | ||
| /// apply the expression rewrite with [`coerce_plan_expr_for_schema`]. | ||
| /// | ||
| /// [`TypeCoercionRewriter::coerce_union`]: https://docs.rs/datafusion-optimizer/latest/datafusion_optimizer/analyzer/type_coercion/struct.TypeCoercionRewriter.html#method.coerce_union |
Contributor
Author
@comphead -- I'm not sure which error message you are referring to? 😅 🙏🏼 |
Contributor
Ignore it, for downstream projects not using the optimizer it would be difficult to understand why UNION crashed, but I believe the documentation is enough |
alamb
reviewed
Aug 26, 2024
| Projection, ScalarUDF, Union, WindowFrame, WindowFrameBound, WindowFrameUnits, | ||
| }; | ||
|
|
||
| /// Performs type coercion by determining the schema |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #12105
Rationale for this change
We construct our own logical plans for a SQL-derivative languages (e.g. InfluxQL). The construction of logic plans using UNION requires that upon construction of the union plan node, we type-coerce the expressions prior to building the rest of the plan. If we don't perform this expression rewrite, then our consuming LIMIT, GROUP BY, ORDER BY, and aggregates will not be built properly.
We do not think our use case is unique, and therefore we are (a) exposing the APIs need to perform this coercion and (b) providing docs.
What changes are included in this PR?
coerce_union_schemaTypeCoercionTypeCoercionRewritercoerce_union_schema.union()API, which does the logical plan construction , as to the how/when/why to use type coercion.Are these changes tested?
No code changes.
Are there any user-facing changes?
No. Only more exposed interfaces to use.