[Feature](datatype) Add IPv4/v6 data type for doris #24965

sjyango · 2023-09-27T08:26:55Z

Proposed changes

Issue Number: close #21370

Describe your changes

Now We do not support IPv4/v6 data type for doris witch is common data type in database
So it's the time to support it ! We have some basic funnctions for data type to check the data type is supported completly

support CURD
support StreamLoad
support Serde functions
support (implicit/explicit) cast functions. The following cast functions have been implemented：

uint8/int8 -> ipv4
uint16/int16 -> ipv4
uint32/int32 -> ipv4
uint64/int64 -> ipv4
string -> ipv4
string -> ipv6

specific functions for itself data type (u can reference clickhouse or mysql)
make a Performance Test result with clickhouse or ElasticSearch in load this data type

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

sjyango · 2023-09-27T08:27:29Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 38. Check the log or trigger a new build to see more.

be/src/olap/types.h

be/src/vec/data_types/data_type_ipv4.cpp

be/src/vec/data_types/data_type_ipv4.h

be/src/vec/data_types/data_type_ipv6.cpp

be/src/vec/data_types/data_type_ipv6.h

doris-robot · 2023-09-27T09:29:49Z

TeamCity be ut coverage result:
Function Coverage: 36.20% (8141/22491)
Line Coverage: 28.41% (65188/229461)
Region Coverage: 27.37% (33809/123530)
Branch Coverage: 24.07% (17288/71816)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0074eb56e95fc6d7597bde9ebf9b82d6f469fc40_0074eb56e95fc6d7597bde9ebf9b82d6f469fc40/report/index.html

doris-robot · 2023-09-27T09:47:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.89 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162311186 Bytes

sjyango · 2023-09-28T01:04:42Z

run p0

amorynan

LGTM

github-actions · 2023-09-28T05:05:48Z

PR approved by anyone and no changes requested.

amorynan · 2023-09-28T05:10:29Z

@xiaokang @yiguolei @morrySnow please review this

morrySnow · 2023-09-28T06:29:55Z

please add rule descriptions for conversions with other types

fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java

sjyango · 2023-09-28T06:35:30Z

please add rule descriptions for conversions with other types

What is the format of rule descriptions for conversions? Are there any relevant examples?

starocean999 · 2023-09-28T07:05:18Z

In general, there are more work need to do for nereids(at least):

Add literal definition of IPV4 and IPV6 like old planner
modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types
modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types
add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

github-actions · 2023-09-28T07:11:12Z

PR approved by at least one committer and no changes requested.

starocean999

In general, there are more work need to do for nereids(at least):

Add literal definition of IPV4 and IPV6 like old planner
modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types
modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types
add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

sjyango · 2023-09-28T07:14:07Z

In general, there are more work need to do for nereids(at least):

Add literal definition of IPV4 and IPV6 like old planner

modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types

modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types

add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

Thank you for reviewing my code, I will complete this work during the National Day!

sjyango · 2023-10-17T13:03:07Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 191. Check the log or trigger a new build to see more.

be/src/olap/types.h

doris-robot · 2023-10-17T14:20:02Z

TeamCity be ut coverage result:
Function Coverage: 36.88% (8282/22457)
Line Coverage: 28.99% (66449/229242)
Region Coverage: 27.67% (34481/124594)
Branch Coverage: 24.35% (17557/72110)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac31834c8057e656af9e9be6c3125c341e7efb04_ac31834c8057e656af9e9be6c3125c341e7efb04/report/index.html

doris-robot · 2023-10-17T14:54:00Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.25 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17161923223 Bytes

sjyango · 2023-10-18T01:32:41Z

run p0

starocean999 · 2023-10-18T02:36:58Z

regression-test/suites/datatype_p0/ip/test_ip_crud.groovy

+    sql """ SET enable_fallback_to_original_planner=false """
+
+    sql """
+        CREATE TABLE test_unique_ip_crud (


for dml statement, even 'SET enable_fallback_to_original_planner=false', the nereids will fallback to old planner if meets any error. So to support ipv4 and ipv6 keyword in nereids, we need modiy the g4 grammer files:
src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
and check visitCreateTable in org/apache/doris/nereids/parser/LogicalPlanBuilder.java works

starocean999 · 2023-10-18T02:52:05Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/IPv6Literal.java

checkValueValid is missing in nereids, shoud it be consistent with old planner?

starocean999 · 2023-10-18T02:52:30Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/IPv4Literal.java

checkValueValid is missing in nereids, shoud it be consistent with old planner?

sjyango · 2023-10-26T02:19:28Z

run buildall

doris-robot · 2023-10-26T03:48:52Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.49 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162053796 Bytes

sjyango · 2023-10-26T05:55:07Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 95. Check the log or trigger a new build to see more.

be/src/vec/runtime/ipv6_value.h

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 71. Check the log or trigger a new build to see more.

be/src/vec/runtime/ipv6_value.h

sjyango · 2023-10-26T06:48:07Z

run clickbench-new

sjyango · 2023-10-26T07:05:39Z

run p0

sjyango · 2023-10-26T07:11:02Z

run p1

sjyango · 2023-10-26T07:18:59Z

run p0

doris-robot · 2023-10-26T08:17:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.18 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162106699 Bytes

github-actions bot reviewed Sep 27, 2023

View reviewed changes

amorynan approved these changes Sep 28, 2023

View reviewed changes

github-actions bot added the reviewed label Sep 28, 2023

starocean999 reviewed Sep 28, 2023

View reviewed changes

fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java Show resolved Hide resolved

starocean999 approved these changes Sep 28, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

This comment was marked as duplicate.

Sign in to view

starocean999 requested changes Sep 28, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

sjyango force-pushed the ip-new branch from 0074eb5 to ac31834 Compare October 17, 2023 13:02

github-actions bot reviewed Oct 17, 2023

View reviewed changes

starocean999 reviewed Oct 18, 2023

View reviewed changes

sjyango added 8 commits October 26, 2023 02:17

add ipv4/v6 codes for nereids optimizer

050900f

update code for nereids

960c27e

restore be/CMakeLists.txt

8e0f586

Prohibition of comparison between IPv4 and IPv6

af9cb70

format be codes

a2810b8

format be codes

4733eb5

format be codes

ca9ac74

resolve conflicts

0b8ae22

sjyango force-pushed the ip-new branch from c92c795 to 50188aa Compare October 26, 2023 02:19

github-actions bot added the meta-change label Oct 26, 2023

resolve master conflicts

958b020

sjyango force-pushed the ip-new branch from 50188aa to 958b020 Compare October 26, 2023 05:47

format be codes

1f9ffcc

github-actions bot reviewed Oct 26, 2023

View reviewed changes

starocean999 approved these changes Oct 26, 2023

View reviewed changes

starocean999 merged commit c1d64a7 into apache:master Oct 26, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023

[Feature](datatype) Add IPv4/v6 data type for doris (apache#24965)

74e63cb

sjyango mentioned this pull request Nov 7, 2023

[Feature](datatype) add be ut codes for IPv4/v6 #26534

Merged

LemonLiTree mentioned this pull request Nov 21, 2023

[Bug] Can't store IPv6 address decimal data for decimal length is not enough #26813

Closed

3 tasks

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[Feature](datatype) Add IPv4/v6 data type for doris (apache#24965)

21eb034

sjyango deleted the ip-new branch January 23, 2024 07:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature](datatype) Add IPv4/v6 data type for doris #24965

[Feature](datatype) Add IPv4/v6 data type for doris #24965

sjyango commented Sep 27, 2023

sjyango commented Sep 27, 2023

github-actions bot left a comment

doris-robot commented Sep 27, 2023

doris-robot commented Sep 27, 2023

sjyango commented Sep 28, 2023

amorynan left a comment

github-actions bot commented Sep 28, 2023

amorynan commented Sep 28, 2023

morrySnow commented Sep 28, 2023

sjyango commented Sep 28, 2023

starocean999 commented Sep 28, 2023 •

edited

Loading

github-actions bot commented Sep 28, 2023

This comment was marked as duplicate.

starocean999 left a comment

sjyango commented Sep 28, 2023

This comment was marked as duplicate.

sjyango commented Oct 17, 2023

github-actions bot left a comment

doris-robot commented Oct 17, 2023

doris-robot commented Oct 17, 2023

sjyango commented Oct 18, 2023

starocean999 Oct 18, 2023

starocean999 Oct 18, 2023

starocean999 Oct 18, 2023

sjyango commented Oct 26, 2023

doris-robot commented Oct 26, 2023

sjyango commented Oct 26, 2023

github-actions bot left a comment

github-actions bot left a comment

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

doris-robot commented Oct 26, 2023

[Feature](datatype) Add IPv4/v6 data type for doris #24965

[Feature](datatype) Add IPv4/v6 data type for doris #24965

Conversation

sjyango commented Sep 27, 2023

Proposed changes

Describe your changes

Further comments

sjyango commented Sep 27, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 27, 2023

doris-robot commented Sep 27, 2023

sjyango commented Sep 28, 2023

amorynan left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 28, 2023

amorynan commented Sep 28, 2023

morrySnow commented Sep 28, 2023

sjyango commented Sep 28, 2023

starocean999 commented Sep 28, 2023 • edited Loading

github-actions bot commented Sep 28, 2023

This comment was marked as duplicate.

starocean999 left a comment

Choose a reason for hiding this comment

sjyango commented Sep 28, 2023

This comment was marked as duplicate.

sjyango commented Oct 17, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Oct 17, 2023

doris-robot commented Oct 17, 2023

sjyango commented Oct 18, 2023

starocean999 Oct 18, 2023

Choose a reason for hiding this comment

starocean999 Oct 18, 2023

Choose a reason for hiding this comment

starocean999 Oct 18, 2023

Choose a reason for hiding this comment

sjyango commented Oct 26, 2023

doris-robot commented Oct 26, 2023

sjyango commented Oct 26, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

sjyango commented Oct 26, 2023

doris-robot commented Oct 26, 2023

starocean999 commented Sep 28, 2023 •

edited

Loading