[Feature](datatype) Add IPv4/v6 data type for doris by sjyango · Pull Request #24965 · apache/doris

sjyango · 2023-09-27T08:26:55Z

Proposed changes

Issue Number: close#21370

Describe your changes

Now We do not support IPv4/v6 data type for doris witch is common data type in database
So it's the time to support it ! We have some basic funnctions for data type to check the data type is supported completly

support CURD
support StreamLoad
support Serde functions
support (implicit/explicit) cast functions. The following cast functions have been implemented：

uint8/int8 -> ipv4
uint16/int16 -> ipv4
uint32/int32 -> ipv4
uint64/int64 -> ipv4
string -> ipv4
string -> ipv6

specific functions for itself data type (u can reference clickhouse or mysql)
make a Performance Test result with clickhouse or ElasticSearch in load this data type

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

sjyango · 2023-09-27T08:27:29Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 38. Check the log or trigger a new build to see more.

doris-robot · 2023-09-27T09:29:49Z

TeamCity be ut coverage result:
Function Coverage: 36.20% (8141/22491)
Line Coverage: 28.41% (65188/229461)
Region Coverage: 27.37% (33809/123530)
Branch Coverage: 24.07% (17288/71816)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0074eb56e95fc6d7597bde9ebf9b82d6f469fc40_0074eb56e95fc6d7597bde9ebf9b82d6f469fc40/report/index.html

doris-robot · 2023-09-27T09:47:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.89 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162311186 Bytes

sjyango · 2023-09-28T01:04:42Z

run p0

amorynan

LGTM

github-actions · 2023-09-28T05:05:48Z

PR approved by anyone and no changes requested.

amorynan · 2023-09-28T05:10:29Z

@xiaokang @yiguolei @morrySnow please review this

morrySnow · 2023-09-28T06:29:55Z

please add rule descriptions for conversions with other types

sjyango · 2023-09-28T06:35:30Z

please add rule descriptions for conversions with other types

What is the format of rule descriptions for conversions? Are there any relevant examples?

starocean999 · 2023-09-28T07:05:18Z

In general, there are more work need to do for nereids(at least):

Add literal definition of IPV4 and IPV6 like old planner
modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types
modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types
add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

github-actions · 2023-09-28T07:11:12Z

PR approved by at least one committer and no changes requested.

starocean999

In general, there are more work need to do for nereids(at least):

Add literal definition of IPV4 and IPV6 like old planner
modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types
modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types
add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

sjyango · 2023-09-28T07:14:07Z

In general, there are more work need to do for nereids(at least):
Add literal definition of IPV4 and IPV6 like old planner
modify fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java for IP types
modify fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java for IP types
add 'set enable_nereids_planner=true' and 'enable_fallback_to_original_planner=false' in groovy regression test cases and verify the result

Thank you for reviewing my code, I will complete this work during the National Day!

sjyango · 2023-10-17T13:03:07Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 191. Check the log or trigger a new build to see more.

doris-robot · 2023-10-17T14:20:02Z

TeamCity be ut coverage result:
Function Coverage: 36.88% (8282/22457)
Line Coverage: 28.99% (66449/229242)
Region Coverage: 27.67% (34481/124594)
Branch Coverage: 24.35% (17557/72110)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac31834c8057e656af9e9be6c3125c341e7efb04_ac31834c8057e656af9e9be6c3125c341e7efb04/report/index.html

doris-robot · 2023-10-17T14:54:00Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.25 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17161923223 Bytes

sjyango · 2023-10-18T01:32:41Z

run p0

starocean999 · 2023-10-18T02:36:58Z

+ sql """ SET enable_fallback_to_original_planner=false """
+
+ sql """
+ CREATE TABLE test_unique_ip_crud (


for dml statement, even 'SET enable_fallback_to_original_planner=false', the nereids will fallback to old planner if meets any error. So to support ipv4 and ipv6 keyword in nereids, we need modiy the g4 grammer files:
src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
and check visitCreateTable in org/apache/doris/nereids/parser/LogicalPlanBuilder.java works

starocean999 · 2023-10-18T02:52:05Z

checkValueValid is missing in nereids, shoud it be consistent with old planner?

starocean999 · 2023-10-18T02:52:30Z

checkValueValid is missing in nereids, shoud it be consistent with old planner?

sjyango · 2023-10-26T02:19:28Z

run buildall

doris-robot · 2023-10-26T03:48:52Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.49 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162053796 Bytes

sjyango · 2023-10-26T05:55:07Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 95. Check the log or trigger a new build to see more.

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 71. Check the log or trigger a new build to see more.

sjyango · 2023-10-26T06:48:07Z

run clickbench-new

sjyango · 2023-10-26T07:05:39Z

run p0

sjyango · 2023-10-26T07:11:02Z

run p1

sjyango · 2023-10-26T07:18:59Z

run p0

doris-robot · 2023-10-26T08:17:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.18 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162106699 Bytes

github-actionsBot reviewed Sep 27, 2023
View reviewed changes

amorynan approved these changes Sep 28, 2023
View reviewed changes

github-actionsBot added the reviewed label Sep 28, 2023

starocean999 reviewed Sep 28, 2023
View reviewed changes

Comment threadfe/fe-common/src/main/java/org/apache/doris/catalog/Type.java

starocean999 approved these changes Sep 28, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

This comment was marked as duplicate.
Sign in to view

starocean999 requested changes Sep 28, 2023
View reviewed changes

This comment was marked as duplicate.
Sign in to view

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 28, 2023

sjyango force-pushed the ip-new branch from 0074eb5 to ac31834Compare October 17, 2023 13:02

github-actionsBot reviewed Oct 17, 2023
View reviewed changes

starocean999 reviewed Oct 18, 2023
View reviewed changes

sjyango added 8 commits October 26, 2023 02:17

add ipv4/v6 codes for nereids optimizer
050900f

update code for nereids
960c27e

restore be/CMakeLists.txt
8e0f586

Prohibition of comparison between IPv4 and IPv6
af9cb70

format be codes
a2810b8

format be codes
4733eb5

format be codes
ca9ac74

resolve conflicts
0b8ae22

sjyango force-pushed the ip-new branch from c92c795 to 50188aaCompare October 26, 2023 02:19

github-actionsBot added the meta-change label Oct 26, 2023

resolve master conflicts
958b020

sjyango force-pushed the ip-new branch from 50188aa to 958b020Compare October 26, 2023 05:47

format be codes
1f9ffcc

github-actionsBot reviewed Oct 26, 2023
View reviewed changes

starocean999 approved these changes Oct 26, 2023
View reviewed changes

starocean999 merged commit c1d64a7 into apache:masterOct 26, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
[Feature](datatype) Add IPv4/v6 data type for doris (apache#24965)
74e63cb

sjyango mentioned this pull request Nov 7, 2023
[Feature](datatype) add be ut codes for IPv4/v6 #26534
Merged

LemonLiTree mentioned this pull request Nov 21, 2023
[Bug] Can't store IPv6 address decimal data for decimal length is not enough #26813
Closed
3 tasks

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
[Feature](datatype) Add IPv4/v6 data type for doris (apache#24965)
21eb034

sjyango deleted the ip-new branch January 23, 2024 07:38

Uh oh!

Conversation

sjyango commented Sep 27, 2023

Proposed changes

Describe your changes

Further comments

Uh oh!

sjyango commented Sep 27, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

sjyango commented Sep 28, 2023

Uh oh!

amorynan left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 28, 2023

Uh oh!

amorynan commented Sep 28, 2023

Uh oh!

morrySnow commented Sep 28, 2023

Uh oh!

Uh oh!

sjyango commented Sep 28, 2023

Uh oh!

starocean999 commented Sep 28, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 28, 2023

Uh oh!

This comment was marked as duplicate.

Uh oh!

starocean999 left a comment

Choose a reason for hiding this comment

Uh oh!

sjyango commented Sep 28, 2023

Uh oh!

This comment was marked as duplicate.

Uh oh!

sjyango commented Oct 17, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Oct 17, 2023

Uh oh!

doris-robot commented Oct 17, 2023

Uh oh!

sjyango commented Oct 18, 2023

Uh oh!

starocean999Oct 18, 2023

Choose a reason for hiding this comment

Uh oh!

starocean999Oct 18, 2023

Choose a reason for hiding this comment

starocean999 commented Sep 28, 2023•
edited
Loading