[Improve](array-func)improve array union support multi params by amorynan · Pull Request #24327 · apache/doris

amorynan · 2023-09-13T13:34:28Z

Proposed changes

Issue Number: close #xxx
Now we just support array union with two arrays

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

amorynan · 2023-09-13T13:39:01Z

run buildall

github-actions · 2023-09-13T13:42:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T14:07:43Z

TeamCity be ut coverage result:
Function Coverage: 36.96% (7968/21556)
Line Coverage: 28.93% (63837/220662)
Region Coverage: 27.87% (33147/118919)
Branch Coverage: 24.43% (16999/69592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d4380de09b436d92f05c0e45bd7ed3e3617387a_7d4380de09b436d92f05c0e45bd7ed3e3617387a/report/index.html

doris-robot · 2023-09-13T14:19:58Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.7 seconds
stream load tsv: 594 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162290689 Bytes

github-actions · 2023-09-14T03:53:53Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2023-09-14T03:54:39Z

run buildall

doris-robot · 2023-09-14T04:12:13Z

TeamCity be ut coverage result:
Function Coverage: 36.96% (7968/21558)
Line Coverage: 28.93% (63848/220721)
Region Coverage: 27.87% (33154/118958)
Branch Coverage: 24.43% (17006/69614)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5caff873b333586c520cb25d9e03c1539bbe9518_5caff873b333586c520cb25d9e03c1539bbe9518/report/index.html

doris-robot · 2023-09-14T04:33:55Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.08 seconds
stream load tsv: 612 seconds loaded 74807831229 Bytes, about 116 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162555795 Bytes

morrySnow · 2023-09-14T06:30:48Z

- [['array_union'], 'ARRAY_VARCHAR', ['ARRAY_VARCHAR', 'ARRAY_VARCHAR'], ''],
- [['array_union'], 'ARRAY_STRING', ['ARRAY_STRING', 'ARRAY_STRING'], ''],
+ [['array_union'], 'ARRAY_BOOLEAN', ['ARRAY_BOOLEAN', 'ARRAY_BOOLEAN', '...'], ''],
+ [['array_union'], 'ARRAY_TINYINT', ['ARRAY_TINYINT', 'ARRAY_TINYINT', '...'], ''],


should update org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayUnion to let it support multi param in Nereids too

cambyzju · 2023-09-14T06:36:26Z

- if (entry.get_mapped() == params.size()) {
+ if constexpr (operation == MapOperation::INTERSECT) {
+ if (entry.get_mapped() == params.size()) {
+ ++dst_off;


same code as line 99~103

I will make micro define

cambyzju · 2023-09-14T06:40:11Z

- if (!set.find(elem)) {
- set.insert(elem);
- return true;
+ void apply(Map& map, const int arg_idx, const int row_idx,


row_idx better use 64bit instead int.
const int == int?

maybe size_t is better ?

xy720

Why calculate multi array union need a map structure? Set is not enough?

amorynan · 2023-09-14T09:52:34Z

Why calculate multi array union need a map structure? Set is not enough?

because MapImpl can apply const ColumnArrayExecutionDatas params, but SetImpl just apply one ColumnArrayExecutionData

github-actions · 2023-09-14T10:00:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-16T00:37:23Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2023-09-16T01:10:50Z

run buildall

doris-robot · 2023-09-16T01:34:08Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (7978/21653)
Line Coverage: 28.86% (64003/221770)
Region Coverage: 27.77% (33194/119515)
Branch Coverage: 24.38% (17040/69900)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4b774ea9e7d3fea3bf616e43c728129500a66705_4b774ea9e7d3fea3bf616e43c728129500a66705/report/index.html

doris-robot · 2023-09-16T01:53:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.27 seconds
stream load tsv: 600 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162510316 Bytes

github-actions · 2023-09-18T02:41:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-18T02:41:05Z

PR approved by anyone and no changes requested.

github-actions · 2023-09-18T09:52:12Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T10:04:09Z

TeamCity be ut coverage result:
Function Coverage: 36.82% (7979/21672)
Line Coverage: 28.84% (64018/221948)
Region Coverage: 27.73% (33224/119799)
Branch Coverage: 24.36% (17029/69916)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2234a7a31488e4c32dff3f1923a4102c20d26d3f_2234a7a31488e4c32dff3f1923a4102c20d26d3f/report/index.html

doris-robot · 2023-09-18T10:17:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.19 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162185221 Bytes

xiaokang · 2023-09-18T12:10:41Z

run buildall

github-actions · 2023-09-18T12:17:21Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T12:27:44Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7981/21672)
Line Coverage: 28.86% (64056/221948)
Region Coverage: 27.75% (33247/119799)
Branch Coverage: 24.38% (17043/69916)
Coverage Report: http://coverage.selectdb-in.cc/coverage/867f06795d2ca7ddb186352839f4ebad7c330ca4_867f06795d2ca7ddb186352839f4ebad7c330ca4/report/index.html

doris-robot · 2023-09-18T12:45:02Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.71 seconds
stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162360025 Bytes

amorynan · 2023-09-19T09:34:20Z

run p0

amorynan · 2023-09-19T09:34:32Z

run p1

amorynan · 2023-09-20T06:04:59Z

@shuke987

github-actions · 2023-09-20T06:20:38Z

PR approved by at least one committer and no changes requested.

xiaokang

LGTM

…#24327)

…#24327)" This reverts commit 6090063.

improve array union support multi params
7d4380d

fix null init:
5caff87

xiaokang added dev/2.0.2 usercase Important user case type label labels Sep 14, 2023

morrySnow reviewed Sep 14, 2023
View reviewed changes

cambyzju reviewed Sep 14, 2023
View reviewed changes

Comment threadbe/src/vec/functions/array/function_array_map.h Outdated

cambyzju reviewed Sep 14, 2023
View reviewed changes

xy720 reviewed Sep 14, 2023
View reviewed changes

Merge branch 'master' into improve-array-union-with-multi-param
99570af

update
4b774ea

morrySnow previously approved these changes Sep 18, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2023

github-actionsBot added the reviewed label Sep 18, 2023

amorynan requested a review from morrySnow September 18, 2023 10:01

Merge branch 'master' into improve-array-union-with-multi-param
867f067

morrySnow approved these changes Sep 20, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2023

xiaokang approved these changes Sep 20, 2023
View reviewed changes

xiaokang merged commit e9435c1 into apache:masterSep 20, 2023

JingDas pushed a commit to JingDas/doris that referenced this pull request Sep 20, 2023
[Improve](array-func)improve array union support multi params (apache…
bf81ad3
…#24327)

xiaokang added merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Sep 20, 2023

xiaokang pushed a commit that referenced this pull request Sep 20, 2023
[Improve](array-func)improve array union support multi params (#24327)
6090063

xiaokang added dev/2.0.2 dev/2.0.2-merged and removed dev/2.0.2-merged dev/2.0.2 labels Sep 20, 2023

xiaokang added a commit that referenced this pull request Sep 20, 2023
Revert "[Improve](array-func)improve array union support multi params (…
cf376a5
…#24327)" This reverts commit 6090063.

xiaokang pushed a commit that referenced this pull request Sep 20, 2023
[Improve](array-func)improve array union support multi params (#24327)
a7715ac

xiaokang mentioned this pull request Sep 30, 2023
Release Note 2.0.2 #25011
Closed

xiaokang mentioned this pull request Dec 3, 2023
Release Note 2.0.3 #27909
Closed

Uh oh!

Conversation

amorynan commented Sep 13, 2023

Proposed changes

Further comments

Uh oh!

amorynan commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 14, 2023

Uh oh!

amorynan commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

doris-robot commented Sep 14, 2023

Uh oh!

morrySnowSep 14, 2023

Choose a reason for hiding this comment

Uh oh!

cambyzjuSep 14, 2023

Choose a reason for hiding this comment

Uh oh!

amorynanSep 15, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cambyzjuSep 14, 2023

Choose a reason for hiding this comment

Uh oh!

amorynanSep 15, 2023

Choose a reason for hiding this comment

Uh oh!

amorynanSep 16, 2023

Choose a reason for hiding this comment

Uh oh!

xy720 left a comment

Choose a reason for hiding this comment

Uh oh!

amorynan commented Sep 14, 2023

Uh oh!

github-actionsBot commented Sep 14, 2023

Uh oh!

github-actionsBot commented Sep 16, 2023

Uh oh!

amorynan commented Sep 16, 2023

Uh oh!

doris-robot commented Sep 16, 2023

Uh oh!

doris-robot commented Sep 16, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

xiaokang commented Sep 18, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

amorynan commented Sep 19, 2023

Uh oh!

amorynan commented Sep 19, 2023

Uh oh!

amorynan commented Sep 20, 2023

Uh oh!