[Enhancement] Support to recognize skip.header.line.count in hive's textfile #47001

Smith-Cruise · 2024-06-14T06:12:28Z

Why I'm doing:

Support skip.header.line.count tblproperties in hive table.

demo:

create table csv_big_skip
(c1 string, c2 int) stored as textfile tblproperties("skip.header.line.count"="10");

What I'm doing:

Support it

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: Smith Cruise <chendingchao1@126.com>

DorianZheng · 2024-06-14T06:40:53Z

be/src/exec/hdfs_scanner_text.cpp

    if (text_file_desc.__isset.compression_type) {
-        _compression_type = CompressionUtils::to_compression_pb(text_file_desc.compression_type);
+        compression_type = CompressionUtils::to_compression_pb(text_file_desc.compression_type);


DorianZheng · 2024-06-14T06:45:23Z

fe/fe-core/src/main/java/com/starrocks/connector/hive/HiveMetastoreApiConverter.java

-        String mapkeyDelim = serdeParams.getOrDefault("mapkey.delim", "");
+        String lineDelim = parameters.getOrDefault(serdeConstants.LINE_DELIM, "");
+        String mapkeyDelim = parameters.getOrDefault(serdeConstants.MAPKEY_DELIM, "");
+        int skipHeaderLineCount = Integer.parseInt(parameters.getOrDefault(serdeConstants.HEADER_COUNT, "0"));


better check if it's legit, such as >= 0

GavinMar · 2024-06-14T07:12:32Z

be/src/exec/hdfs_scanner_text.h

+    Status _create_csv_reader();
+    Status _setup_compression_type(const TTextFileDesc& text_file_desc);
+    Status _setup_delimiter(const TTextFileDesc& text_file_desc);
+    StatusOr<bool> _has_uft8_bom() const;


typo for utf8?

Signed-off-by: Smith Cruise <chendingchao1@126.com>

sonarqubecloud · 2024-06-14T09:56:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-06-14T12:12:34Z

[FE Incremental Coverage Report]

✅ pass : 21 / 22 (95.45%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/connector/hive/HiveMetastoreApiConverter.java	14	15	93.33%	[594]
🔵	com/starrocks/connector/hive/TextFileFormatDesc.java	7	7	100.00%	[]

github-actions · 2024-06-14T12:18:47Z

[BE Incremental Coverage Report]

✅ pass : 57 / 57 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/exec/hdfs_scanner.cpp	6	6	100.00%	[]
🔵	be/src/exec/hdfs_scanner_text.cpp	51	51	100.00%	[]

trueeyu · 2024-06-17T03:39:39Z

be/src/exec/hdfs_scanner.cpp

@@ -627,4 +628,12 @@ void HdfsScanner::move_split_tasks(std::vector<pipeline::ScanSplitContextPtr>* s
    }
 }

+CompressionTypePB HdfsScanner::get_compression_type_from_path(const std::string& filename) {


Duplicate with the function return_compression_type_from_filename of hdf_scanner_text.cpp ?

Make it to one common function?

trueeyu · 2024-06-17T04:37:55Z

gensrc/thrift/Descriptors.thrift

@@ -100,6 +100,8 @@ struct TTextFileDesc {

    // escape character
    8: optional i8 escape
+
+    9: optional i32 skip_header_line_count


After adding this Skip, will it cause problems with the optimizer, such as Limit 1. In the past, FE would select 1 file, but with this skip, problems may occur?

github-actions · 2024-06-17T04:45:13Z

@Mergifyio backport branch-3.3

github-actions · 2024-06-17T04:45:14Z

@Mergifyio backport branch-3.2

github-actions · 2024-06-17T04:45:15Z

@Mergifyio backport branch-3.1

mergify · 2024-06-17T04:45:25Z

backport branch-3.3

✅ Backports have been created

#47078 [Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) has been created for branch branch-3.3

mergify · 2024-06-17T04:45:26Z

backport branch-3.2

✅ Backports have been created

#47079 [Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) has been created for branch branch-3.2 but encountered conflicts

mergify · 2024-06-17T04:45:27Z

backport branch-3.1

✅ Backports have been created

#47080 [Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) has been created for branch branch-3.1 but encountered conflicts

…extfile (#47001) Signed-off-by: Smith Cruise <chendingchao1@126.com> (cherry picked from commit 2c838f7)

…extfile (#47001) Signed-off-by: Smith Cruise <chendingchao1@126.com> (cherry picked from commit 2c838f7) # Conflicts: # be/src/exec/hdfs_scanner_text.cpp # be/src/exec/hdfs_scanner_text.h # fe/fe-core/src/main/java/com/starrocks/connector/hive/TextFileFormatDesc.java # fe/fe-core/src/test/java/com/starrocks/connector/hive/HiveMetaClientTest.java # fe/fe-core/src/test/java/com/starrocks/connector/hive/TextFileFormatDescTest.java

…extfile (#47001) Signed-off-by: Smith Cruise <chendingchao1@126.com> (cherry picked from commit 2c838f7) # Conflicts: # be/src/exec/hdfs_scanner.cpp # be/src/exec/hdfs_scanner_text.cpp # be/src/exec/hdfs_scanner_text.h # be/test/exec/hdfs_scanner_test.cpp # fe/fe-core/src/main/java/com/starrocks/connector/hive/HiveMetastoreApiConverter.java # fe/fe-core/src/main/java/com/starrocks/connector/hive/TextFileFormatDesc.java # fe/fe-core/src/test/java/com/starrocks/connector/hive/HiveMetaClientTest.java # fe/fe-core/src/test/java/com/starrocks/connector/hive/TextFileFormatDescTest.java

…extfile (backport #47001) (#47078) Co-authored-by: Smith Cruise <chendingchao1@126.com>

…er.line.count in hive's textfile (backport #47001) (#47082) Signed-off-by: Smith Cruise <chendingchao1@126.com>

… skip.header.line.count in hive's textfile (backport StarRocks#47001) (StarRocks#47082) Signed-off-by: Smith Cruise <chendingchao1@126.com>

Smith-Cruise added 3 commits June 13, 2024 19:29

fix fe

2781e5c

Signed-off-by: Smith Cruise <chendingchao1@126.com>

done be part

ef8d5e1

Signed-off-by: Smith Cruise <chendingchao1@126.com>

done

9ab770c

Signed-off-by: Smith Cruise <chendingchao1@126.com>

Smith-Cruise requested review from a team as code owners June 14, 2024 06:12

wanpengfei-git added the PROTO-REVIEW label Jun 14, 2024

mergify bot assigned Smith-Cruise Jun 14, 2024

wanpengfei-git requested a review from a team June 14, 2024 06:13

Smith-Cruise changed the title ~~[Feature] Support to recognize skip.header.line.count in hive's textfile~~ [Enhancement] Support to recognize skip.header.line.count in hive's textfile Jun 14, 2024

github-actions bot added 3.3 3.2 3.1 labels Jun 14, 2024

format

0111a50

Signed-off-by: Smith Cruise <chendingchao1@126.com>

DorianZheng reviewed Jun 14, 2024

View reviewed changes

GavinMar reviewed Jun 14, 2024

View reviewed changes

improve

3ff45ae

Signed-off-by: Smith Cruise <chendingchao1@126.com>

GavinMar approved these changes Jun 17, 2024

View reviewed changes

DorianZheng approved these changes Jun 17, 2024

View reviewed changes

trueeyu reviewed Jun 17, 2024

View reviewed changes

trueeyu approved these changes Jun 17, 2024

View reviewed changes

trueeyu merged commit 2c838f7 into StarRocks:main Jun 17, 2024
61 of 63 checks passed

github-actions bot removed the 3.3 label Jun 17, 2024

github-actions bot removed 3.2 3.1 labels Jun 17, 2024

mergify bot pushed a commit that referenced this pull request Jun 17, 2024

[Enhancement] Support to recognize skip.header.line.count in hive's t…

8266ba7

…extfile (#47001) Signed-off-by: Smith Cruise <chendingchao1@126.com> (cherry picked from commit 2c838f7)

mergify bot mentioned this pull request Jun 17, 2024

[Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) #47078

Merged

42 tasks

mergify bot mentioned this pull request Jun 17, 2024

[Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) #47079

Closed

42 tasks

mergify bot mentioned this pull request Jun 17, 2024

[Enhancement] Support to recognize skip.header.line.count in hive's textfile (backport #47001) #47080

Closed

42 tasks

Smith-Cruise deleted the csv-skip-header branch June 17, 2024 05:01

Smith-Cruise mentioned this pull request Jun 17, 2024

[Enhancement][Cherry-Pick][Branch-3.2] Support to recognize skip.header.line.count in hive's textfile (backport #47001) #47082

Merged

24 tasks

wanpengfei-git pushed a commit that referenced this pull request Jun 17, 2024

[Enhancement] Support to recognize skip.header.line.count in hive's t…

ac43bd7

…extfile (backport #47001) (#47078) Co-authored-by: Smith Cruise <chendingchao1@126.com>

github-actions bot added the 3.3-merged label Jun 17, 2024

packy92 pushed a commit that referenced this pull request Jun 17, 2024

[Enhancement][Cherry-Pick][Branch-3.2] Support to recognize skip.head…

6245af5

…er.line.count in hive's textfile (backport #47001) (#47082) Signed-off-by: Smith Cruise <chendingchao1@126.com>

github-actions bot added 3.2-merged 3.3 3.2 labels Jun 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Support to recognize skip.header.line.count in hive's textfile #47001

[Enhancement] Support to recognize skip.header.line.count in hive's textfile #47001

Smith-Cruise commented Jun 14, 2024 •

edited

Loading

DorianZheng Jun 14, 2024

DorianZheng Jun 14, 2024

Smith-Cruise Jun 14, 2024

GavinMar Jun 14, 2024

Smith-Cruise Jun 14, 2024

sonarqubecloud bot commented Jun 14, 2024

github-actions bot commented Jun 14, 2024

github-actions bot commented Jun 14, 2024

trueeyu Jun 17, 2024

trueeyu Jun 17, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

mergify bot commented Jun 17, 2024 •

edited

Loading

mergify bot commented Jun 17, 2024 •

edited

Loading

mergify bot commented Jun 17, 2024 •

edited

Loading

[Enhancement] Support to recognize skip.header.line.count in hive's textfile #47001

[Enhancement] Support to recognize skip.header.line.count in hive's textfile #47001

Conversation

Smith-Cruise commented Jun 14, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

DorianZheng Jun 14, 2024

Choose a reason for hiding this comment

DorianZheng Jun 14, 2024

Choose a reason for hiding this comment

Smith-Cruise Jun 14, 2024

Choose a reason for hiding this comment

GavinMar Jun 14, 2024

Choose a reason for hiding this comment

Smith-Cruise Jun 14, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Jun 14, 2024

Quality Gate passed

github-actions bot commented Jun 14, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Jun 14, 2024

[BE Incremental Coverage Report]

file detail

trueeyu Jun 17, 2024

Choose a reason for hiding this comment

trueeyu Jun 17, 2024

Choose a reason for hiding this comment

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

github-actions bot commented Jun 17, 2024

mergify bot commented Jun 17, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Jun 17, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Jun 17, 2024 • edited Loading

✅ Backports have been created

Smith-Cruise commented Jun 14, 2024 •

edited

Loading

mergify bot commented Jun 17, 2024 •

edited

Loading

mergify bot commented Jun 17, 2024 •

edited

Loading

mergify bot commented Jun 17, 2024 •

edited

Loading