Add standalone example of using the SQL frontend#11088
Merged
alamb merged 11 commits intoapache:mainfrom Jul 2, 2024
Merged
Conversation
Contributor
Author
|
@andygrove or @jdye64 I wonder if you might have some time to review this PR as I think it is relevant to to your use of DataFusion |
github-actions
bot
removed
the
documentation
11 tasks
Contributor
Author
|
I am working on getting the CI to run successfully on this PR, see #11173 |
Contributor
Author
|
🤔 this might also make https://github.com/apache/datafusion/blob/main/datafusion/sql/examples/sql.rs redundant |
mustafasrepo
reviewed
Jul 2, 2024
| // The initial LogicalPlan is a mechanical translation from the parsed SQL | ||
| // and often can not run without the Analyzer passes. | ||
| // | ||
| // In this example, `person.age` is actually a different data type (Int32) |
Contributor
There was a problem hiding this comment.
At the source schema data type defined as Uint8
mustafasrepo
reviewed
Jul 2, 2024
| // (Int64). However, this plan is not as efficient as it could be, as it | ||
| // will require casting *each row* of the input to UInt64 before comparison | ||
| // to 21 and 32. To optimize this query's performance, it is better to cast | ||
| // the constants once at plan time to Int32. |
Contributor
There was a problem hiding this comment.
I guess this should be to Uint8
mustafasrepo
approved these changes
Jul 2, 2024
Contributor
mustafasrepo
left a comment
mustafasrepo
left a comment
There was a problem hiding this comment.
Thanks @alamb for this PR.
comphead
reviewed
Jul 2, 2024
| let context_provider = MyContextProvider::default(); | ||
| let sql_to_rel = SqlToRel::new(&context_provider); | ||
| let logical_plan = sql_to_rel.sql_statement_to_plan(statements[0].clone())?; | ||
| println!( |
Contributor
There was a problem hiding this comment.
maybe we can use assert! here to let the user know the expected output?
Contributor
Author
There was a problem hiding this comment.
that is a good idea -- I will do so (the existing examples are inconsistent with that).
comphead
reviewed
Jul 2, 2024
| config.options(), | ||
| observe_analyzer, | ||
| )?; | ||
| println!( |
comphead
reviewed
Jul 2, 2024
| // 4. Pushed the filter into the scan | ||
| let optimized_plan = | ||
| Optimizer::new().optimize(analyzed_plan, &config, observe_optimizer)?; | ||
| println!( |
comphead
reviewed
Jul 2, 2024
comphead
approved these changes
Jul 2, 2024
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Contributor
Author
|
Thanks everyone for the comments and feedback |
comphead
added a commit
to comphead/arrow-datafusion
that referenced
this pull request
Jul 8, 2024
* Add standalone example of using the SQL frontend * Disable debug info for all example jobs * Revert "Disable debug info for all example jobs" This reverts commit f222a10. * fix type * Int32 --> Int8 * Use assert_eq rather than println * Update datafusion-examples/examples/sql_frontend.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
* Add standalone example of using the SQL frontend * Disable debug info for all example jobs * Revert "Disable debug info for all example jobs" This reverts commit f222a10. * fix type * Int32 --> Int8 * Use assert_eq rather than println * Update datafusion-examples/examples/sql_frontend.rs Co-authored-by: Oleks V <comphead@users.noreply.github.com> --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #10855
Rationale for this change
The current
rewrite_expr.rsexample has three distinct examples:AnalyzerRuleOptimizerRuleEach I think should be its own example to make them easier to find and more full featured
What changes are included in this PR?
sql_frontend.rsAre these changes tested?
Yes by CI
Are there any user-facing changes?