Skip to content

[core] Fix Identifier should parse backquote. #5390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

LinMingQiang
Copy link
Contributor

When using flink procedure, table name can not parsed correctly when use `` quote.

But Spark request use `` for SYSTEM_TABLE_SPLITTER ($) . So this is incompatible with Spark.

e.g :

Flink is OK But Spark is ERR  :  CALL sys.create_branch('default.T$branch_branch1', ......)

Flink is ERR But Spark is OK :  CALL sys.create_branch('default.`T$branch_branch1`', ......)

Flink will parse `T$branch_branch1` to table is `T  and branch is branch_branch1`.

image

Why :

Flink use org.apache.paimon.catalog.Identifier to parse the tableName and miss processing backquote,
spark use spark.sessionState().sqlParser().parseMultipartIdentifier to parse.

Purpose

Linked issue: close #xxx

Tests

API and Format

Documentation

@liyubin117
Copy link
Contributor

liyubin117 commented Apr 3, 2025

I doubt whether the feature is necessary, because Flink regards `example.view` as a whole table name differ from Spark. refer to https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/common/#expanding-table-identifiers
We should follow the corresponding syntax rules.

On the other hand, the feature cannot deal with legal default.`T`$branch_branch_A.

@LinMingQiang
Copy link
Contributor Author

I doubt whether the feature is necessary, because Flink regards example.view as a whole table name differ from Spark. refer to https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/common/#expanding-table-identifiers We should follow the corresponding syntax rules.

On the other hand, the feature cannot deal with legal default.`T`$branch_branch_A.

Thanks for your comment. But i think this has nothing to do with flink.

CALL sys.create_branch('default.`table`$branch_branch1', ......) should be executed successfully in flink.

And i will fix this case default.`T`$branch_branch_A .

@liyubin117
Copy link
Contributor

liyubin117 commented Apr 7, 2025

Thanks for your reply, we may need to consider the following scenairos.
image
image

@LinMingQiang
Copy link
Contributor Author

Thanks for your reply, we may need to consider the following scenairos. image image

In your example, you create a table called table`#branched. This is no problem in Flink-SQL parsing engine, as the backticks are handled correctly. But it is not processed in procedure, so the correct table name table`#branched cannot be parsed. This is also the purpose of this PR, here we can't just do a simple replacement, we need a parser to parse the table name like SparkUtils#catalogAndIdentifier does .

@gmdfalk
Copy link

gmdfalk commented Apr 7, 2025

Thanks for your reply, we may need to consider the following scenairos. image image

In your example, you create a table called table#branched. This is no problem in Flink-SQL parsing engine, as the backticks are handled correctly. But it is not processed in procedure, so the correct table name table#branched cannot be parsed. This is also the purpose of this PR, here we can't just do a simple replacement, we need a parser to parse the table name like SparkUtils#catalogAndIdentifier does .

I believe you also cannot do a simple replacement like currently if you want to support any entity (catalog, database or table) with special characters in it.
For instance if the database name contains a dot:

CALL sys.compact(`table` => '`my_catalog`.`my_database.containing_dot`.my_table');

@LinMingQiang
Copy link
Contributor Author

Thanks for your reply, we may need to consider the following scenairos. image image

In your example, you create a table called table#branched. This is no problem in Flink-SQL parsing engine, as the backticks are handled correctly. But it is not processed in procedure, so the correct table name table#branched cannot be parsed. This is also the purpose of this PR, here we can't just do a simple replacement, we need a parser to parse the table name like SparkUtils#catalogAndIdentifier does .

I believe you also cannot do a simple replacement like currently if you want to support any entity (catalog, database or table) with special characters in it. For instance if the database name contains a dot:

CALL sys.compact(`table` => '`my_catalog`.`my_database.containing_dot`.my_table');

Sure, @JingsongLi WDYT. Do we need to implement a parser to parse the table name like spark procedure do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants