Skip to content

chore(deps): Update sqlparser to 0.55.0 #15183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Mar 24, 2025
Merged

chore(deps): Update sqlparser to 0.55.0 #15183

merged 14 commits into from
Mar 24, 2025

Conversation

PokIsemaine
Copy link
Contributor

@PokIsemaine PokIsemaine commented Mar 12, 2025

Which issue does this PR close?

Rationale for this change

Keep up with latest SQL parser dependencies

What changes are included in this PR?

  1. Update to sqlparser to latest released
  2. Update DataFusion to use the latest APIs

Note

  1. JoinOperator: Due to multiple JoinOperator mappings to the same JoinType, some tests need to be modified
JoinOperator::Inner(constraint) | JoinOperator::Join(constraint) => {
    self.parse_join(left, right, constraint, JoinType::Inner, planner_context)
}
  1. OrderBy: Add support for ORDER BY ALL datafusion-sqlparser-rs#1724
    I will create an issue for ORDER BY ALL and implement it later. Like GROUP BY ALL, this requires obtaining select_exprs.

  2. struct.slt: user => user_infomation. I found that in the struct, user is parsed as a Function rather than an Ident.
    feat: Parse special keywords as functions (current_user, user, etc) datafusion-sqlparser-rs#561

  3. update.slt: sqlparser already supports multiple tables in the UPDATE FROM clause, but DataFusion does not yet support it. Support multiple tables in UPDATE FROM clause datafusion-sqlparser-rs#1681

ref

  1. Changelog: https://github.com/apache/datafusion-sqlparser-rs/blob/main/changelog/0.55.0.md
  2. SQLExpr::Case: Replace parallel condition/result vectors with single CaseWhen vector in Expr::Case datafusion-sqlparser-rs#1733
  3. ColumnDef Collation: Treat COLLATE like any other column option datafusion-sqlparser-rs#1731
  4. DataType: Parse SIGNED INTEGER type in MySQL CAST datafusion-sqlparser-rs#1739
  5. EXECUTE: Add support for EXECUTE IMMEDIATE datafusion-sqlparser-rs#1717
  6. CreateTable: Add support for Create Iceberg Table statement for Snowflake parser datafusion-sqlparser-rs#1664
  7. QualifiedWildcard BigQuery: Add support for select expr star datafusion-sqlparser-rs#1680
  8. TypedString: Parse SIGNED INTEGER type in MySQL CAST datafusion-sqlparser-rs#1739
  9. SelectFlavor: Implement FROM-first selects datafusion-sqlparser-rs#1713
  10. StartTransaction: BigQuery: Add support for BEGIN datafusion-sqlparser-rs#1718

Are these changes tested?

Yes by CI

Are there any user-facing changes?

@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions labels Mar 12, 2025
@github-actions github-actions bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate ffi Changes to the ffi crate labels Mar 13, 2025
@github-actions github-actions bot removed documentation Improvements or additions to documentation development-process Related to development process of DataFusion physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate catalog Related to the catalog crate common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate labels Mar 13, 2025
@github-actions github-actions bot removed the ffi Changes to the ffi crate label Mar 13, 2025
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Mar 15, 2025
@PokIsemaine PokIsemaine marked this pull request as ready for review March 15, 2025 15:38
@alamb
Copy link
Contributor

alamb commented Mar 17, 2025

Amazing! Thank you @PokIsemaine -- I'll try and review this sometime this week

FYI @jonahgao

@jonahgao jonahgao self-requested a review March 17, 2025 15:25
Copy link
Member

@jonahgao jonahgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @PokIsemaine. I reviewed part of the changes. I think some unwraps can be replaced with errors to prevent panic.

@@ -216,8 +216,7 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
pub(super) fn sql_case_identifier_to_expr(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to this PR, why is it called case_identifier🤔
I'm not sure what the connection is between case and identifier.

Copy link
Member

@jonahgao jonahgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think we can merge this as it is and improve it in follow-up PRs.

.map(|object_name_part| {
object_name_part
.as_ident()
.map_or_else(String::new, ident_to_string)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to return an error than to silently use a default value.

@github-actions github-actions bot added the datasource Changes to the datasource crate label Mar 22, 2025
@jonahgao jonahgao merged commit 5210a2b into apache:main Mar 24, 2025
29 checks passed
zhuqi-lucas pushed a commit to zhuqi-lucas/arrow-datafusion that referenced this pull request Mar 24, 2025
* chore(deps): Update sqlparser to 0.55.0

* merge upstream

* style: cargo fmt

* fix: import

* style: cargo fmt

* fix: struct.slt, update.slt and lateral join

* chore: cargo fmt & remove unwrap

* fix: remove meaningless comment for rustfmt

* refactor: plan_datafusion_err, remove clone

* remove println

---------

Co-authored-by: silezhou <zhousile2002@gmail.com>
Co-authored-by: jonahgao <jonahgao@msn.com>
@alamb
Copy link
Contributor

alamb commented Mar 24, 2025

Thank you so much @jonahgao and @PokIsemaine

qstommyshu pushed a commit to qstommyshu/datafusion that referenced this pull request Mar 27, 2025
* chore(deps): Update sqlparser to 0.55.0

* merge upstream

* style: cargo fmt

* fix: import

* style: cargo fmt

* fix: struct.slt, update.slt and lateral join

* chore: cargo fmt & remove unwrap

* fix: remove meaningless comment for rustfmt

* refactor: plan_datafusion_err, remove clone

* remove println

---------

Co-authored-by: silezhou <zhousile2002@gmail.com>
Co-authored-by: jonahgao <jonahgao@msn.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datasource Changes to the datasource crate logical-expr Logical plan and expressions sql SQL Planner sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to sqlparser 0.55.0
3 participants