forked from apache/doris
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[fix](Tvf) return empty set when tvf queries an empty file or an erro…
…r uri (apache#25280) ### Before: return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. ### Now: we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ```
- Loading branch information
Showing
13 changed files
with
256 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 6 additions & 0 deletions
6
regression-test/data/external_table_p0/tvf/test_hdfs_tvf_error_uri.out
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !select1 -- | ||
|
|
||
| -- !desc1 -- | ||
| __dummy_col TEXT Yes false \N NONE | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !select -- | ||
|
|
||
| -- !desc -- | ||
| __dummy_col TEXT Yes false \N NONE | ||
|
|
||
| -- !select2 -- | ||
| 1 doris 18 | ||
| 2 nereids 20 | ||
| 3 xxx 22 | ||
| 4 yyy 21 | ||
|
|
||
| -- !des2 -- | ||
| c1 TEXT Yes false \N NONE | ||
| c2 TEXT Yes false \N NONE | ||
| c3 TEXT Yes false \N NONE | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !select -- | ||
|
|
||
| -- !desc -- | ||
| __dummy_col TEXT Yes false \N NONE | ||
|
|
||
| -- !select2 -- | ||
|
|
||
| -- !desc2 -- | ||
| __dummy_col TEXT Yes false \N NONE | ||
|
|
43 changes: 43 additions & 0 deletions
43
regression-test/suites/external_table_p0/tvf/test_hdfs_tvf_error_uri.groovy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| suite("test_hdfs_tvf_error_uri","external,hive,tvf,external_docker") { | ||
| String hdfs_port = context.config.otherConfigs.get("hdfs_port") | ||
| String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") | ||
|
|
||
| // It's okay to use random `hdfsUser`, but can not be empty. | ||
| def hdfsUserName = "doris" | ||
| def format = "csv" | ||
| def defaultFS = "hdfs://${externalEnvIp}:${hdfs_port}" | ||
| def uri = "" | ||
|
|
||
| String enabled = context.config.otherConfigs.get("enableHiveTest") | ||
| if (enabled != null && enabled.equalsIgnoreCase("true")) { | ||
| // test csv format | ||
| uri = "${defaultFS}" + "/user/doris/preinstalled_data/csv_format_test/no_exist_file.csv" | ||
| format = "csv" | ||
| order_qt_select1 """ select * from HDFS( | ||
| "uri" = "${uri}", | ||
| "hadoop.username" = "${hdfsUserName}", | ||
| "format" = "${format}"); """ | ||
|
|
||
| order_qt_desc1 """ desc function HDFS( | ||
| "uri" = "${uri}", | ||
| "hadoop.username" = "${hdfsUserName}", | ||
| "format" = "${format}"); """ | ||
| } | ||
| } |
69 changes: 69 additions & 0 deletions
69
regression-test/suites/load_p0/tvf/test_tvf_empty_file.groovy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| suite("test_tvf_empty_file", "p0") { | ||
| String ak = getS3AK() | ||
| String sk = getS3SK() | ||
| String s3_endpoint = getS3Endpoint() | ||
| String region = getS3Region() | ||
| String bucket = context.config.otherConfigs.get("s3BucketName"); | ||
|
|
||
| String path = "regression/datalake" | ||
|
|
||
| // ${path}/empty_file_test.csv is an empty file | ||
| // so it should return empty sets. | ||
| order_qt_select """ SELECT * FROM S3 ( | ||
| "uri" = "http://${bucket}.${s3_endpoint}/${path}/empty_file_test.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
|
|
||
| order_qt_desc """ desc function S3 ( | ||
| "uri" = "http://${bucket}.${s3_endpoint}/${path}/empty_file_test.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
|
|
||
| // ${path}/empty_file_test*.csv matches 3 files: | ||
| // empty_file_test.csv, empty_file_test_1.csv, empty_file_test_2.csv | ||
| // empty_file_test.csv is an empty file, but | ||
| // empty_file_test_1.csv and empty_file_test_2.csv have data | ||
| // so it should return data of empty_file_test_1.csv and empty_file_test_2.cs | ||
| order_qt_select2 """ SELECT * FROM S3 ( | ||
| "uri" = "http://${bucket}.${s3_endpoint}/${path}/empty_file_test*.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ) order by c1; | ||
| """ | ||
|
|
||
| order_qt_des2 """ desc function S3 ( | ||
| "uri" = "http://${bucket}.${s3_endpoint}/${path}/empty_file_test*.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
| } |
61 changes: 61 additions & 0 deletions
61
regression-test/suites/load_p0/tvf/test_tvf_error_url.groovy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| suite("test_tvf_error_url", "p0") { | ||
| String ak = getS3AK() | ||
| String sk = getS3SK() | ||
| String s3_endpoint = getS3Endpoint() | ||
| String region = getS3Region() | ||
| String bucket = context.config.otherConfigs.get("s3BucketName"); | ||
|
|
||
| String path = "select_tvf/no_exists_file_test" | ||
| order_qt_select """ SELECT * FROM S3 ( | ||
| "uri" = "http://${s3_endpoint}/${bucket}/${path}/no_exist_file1.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
|
|
||
| order_qt_desc """ desc function S3 ( | ||
| "uri" = "http://${s3_endpoint}/${bucket}/${path}/no_exist_file1.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
|
|
||
| order_qt_select2 """ SELECT * FROM S3 ( | ||
| "uri" = "http://${s3_endpoint}/${bucket}/${path}/*.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
|
|
||
| order_qt_desc2 """ desc function S3 ( | ||
| "uri" = "http://${s3_endpoint}/${bucket}/${path}/*.csv", | ||
| "ACCESS_KEY"= "${ak}", | ||
| "SECRET_KEY" = "${sk}", | ||
| "format" = "csv", | ||
| "region" = "${region}" | ||
| ); | ||
| """ | ||
| } |