[feature](merge-on-write) enable merge-on-write by default by bobhan1 · Pull Request #27188 · apache/doris

bobhan1 · 2023-11-17T07:46:11Z

Proposed changes

make the default implementation to merge on write for unique tables.
add docs for table property enable_unique_key_merge_on_write.
display the property enable_unique_key_merge_on_write in show create table command for unique table regardless the value of the property.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

bobhan1 · 2023-11-17T08:03:03Z

run buildall

bobhan1 · 2023-11-17T09:09:24Z

run buildall

bobhan1 · 2023-11-17T09:24:26Z

run buildall

doris-robot · 2023-11-17T10:06:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.38 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17099940833 Bytes

bobhan1 · 2023-11-17T10:24:21Z

run buildall

zhannngchen · 2023-11-17T10:27:06Z

you can refer to my previous PR: #19017

doris-robot · 2023-11-17T10:57:43Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.32 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17099699651 Bytes

doris-robot · 2023-11-17T11:13:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.41 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17100688470 Bytes

bobhan1 · 2023-11-17T11:24:15Z

run buildall

bobhan1 · 2023-11-17T11:26:10Z

run buildall

doris-robot · 2023-11-17T11:29:19Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.81 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17098680756 Bytes

doris-robot · 2023-11-17T11:59:40Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.02 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17099983166 Bytes

doris-robot · 2023-11-17T12:03:16Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.67 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17158456047 Bytes

bobhan1 · 2023-11-20T02:49:58Z

run buildall

doris-robot · 2023-11-20T04:54:01Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.01 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17099573417 Bytes

doris-robot · 2023-11-20T04:58:44Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit db91e487dd3a6749f3962cbb6653459a86aa9f5a, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4958	4687	4655	4655
q2	360	147	159	147
q3	2039	1976	1927	1927
q4	1395	1281	1242	1242
q5	3951	3943	4033	3943
q6	256	130	130	130
q7	1397	878	902	878
q8	2783	2782	2785	2782
q9	9899	9509	9343	9343
q10	3457	3519	3534	3519
q11	377	242	257	242
q12	440	300	291	291
q13	4574	3853	3825	3825
q14	320	293	282	282
q15	582	534	518	518
q16	671	586	589	586
q17	1132	923	920	920
q18	8005	7551	7371	7371
q19	1664	1683	1688	1683
q20	579	332	325	325
q21	4447	4035	4008	4008
q22	483	371	369	369
Total cold run time: 53769 ms
Total hot run time: 48986 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4582	4573	4589	4573
q2	330	223	247	223
q3	4010	4011	4026	4011
q4	2742	2725	2726	2725
q5	9625	9698	9611	9611
q6	252	121	125	121
q7	3025	2543	2504	2504
q8	4395	4417	4408	4408
q9	12991	12810	12987	12810
q10	4027	4158	4175	4158
q11	798	654	691	654
q12	971	818	824	818
q13	4297	3553	3534	3534
q14	391	346	353	346
q15	554	513	516	513
q16	750	683	670	670
q17	3916	3889	3894	3889
q18	9645	8908	9215	8908
q19	1777	1784	1774	1774
q20	2404	2034	2067	2034
q21	8870	8720	8724	8720
q22	882	826	812	812
Total cold run time: 81234 ms
Total hot run time: 77816 ms

bobhan1 · 2023-11-21T07:34:57Z

run buildall

doris-robot · 2023-11-21T08:27:42Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.77 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17098291592 Bytes

doris-robot · 2023-11-21T08:54:01Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 1bfadb1f21cb6166c4ebc13c6c00c30043f15215, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4913	4646	4652	4646
q2	369	144	140	140
q3	2028	1901	1910	1901
q4	1382	1253	1262	1253
q5	3929	3944	4018	3944
q6	240	132	132	132
q7	1413	897	890	890
q8	2740	2772	2756	2756
q9	18425	13553	9666	9666
q10	13171	3540	3521	3521
q11	394	254	247	247
q12	466	293	297	293
q13	21850	3784	3806	3784
q14	321	293	291	291
q15	590	539	527	527
q16	669	589	567	567
q17	1151	969	925	925
q18	7843	7269	7370	7269
q19	1663	1690	1673	1673
q20	535	306	308	306
q21	4435	3935	3937	3935
q22	464	362	366	362
Total cold run time: 88991 ms
Total hot run time: 49028 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4601	4576	4571	4571
q2	342	235	258	235
q3	4014	3994	3990	3990
q4	2707	2696	2713	2696
q5	9823	9687	9657	9657
q6	245	122	127	122
q7	2984	2457	2504	2457
q8	4440	4424	4397	4397
q9	13230	13122	13127	13122
q10	4101	4192	4216	4192
q11	813	711	648	648
q12	970	815	831	815
q13	4275	3598	3595	3595
q14	389	335	354	335
q15	572	520	531	520
q16	733	709	676	676
q17	3902	3892	3875	3875
q18	9510	9140	8896	8896
q19	1768	1782	1796	1782
q20	2411	2060	2033	2033
q21	8755	8622	8843	8622
q22	877	742	807	742
Total cold run time: 81462 ms
Total hot run time: 77978 ms

bobhan1 · 2023-11-21T09:02:01Z

run buildall

doris-robot · 2023-11-21T11:01:50Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit fc3adfa85180dd376129be3abc3ea0f53c67dbc5, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4890	4640	4649	4640
q2	359	155	160	155
q3	2021	1881	1902	1881
q4	1416	1252	1217	1217
q5	3915	3904	3987	3904
q6	241	128	132	128
q7	1425	888	910	888
q8	2736	2755	2747	2747
q9	57773	9561	9671	9561
q10	12987	3521	3518	3518
q11	372	241	250	241
q12	1679	289	290	289
q13	20478	3802	3784	3784
q14	315	290	285	285
q15	588	540	536	536
q16	664	574	580	574
q17	1136	991	941	941
q18	7790	7283	7349	7283
q19	1717	1688	1666	1666
q20	555	292	307	292
q21	7412	3957	3983	3957
q22	460	366	379	366
Total cold run time: 130929 ms
Total hot run time: 48853 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4672	4578	4571	4571
q2	339	232	278	232
q3	3988	3967	3969	3967
q4	2687	2668	2684	2668
q5	9753	9632	9637	9632
q6	238	119	120	119
q7	3002	2455	2474	2455
q8	4413	4438	4438	4438
q9	13145	13070	13139	13070
q10	4104	4180	4205	4180
q11	778	649	641	641
q12	971	800	829	800
q13	4266	3601	3622	3601
q14	374	332	362	332
q15	579	525	524	524
q16	728	691	676	676
q17	3841	3823	3931	3823
q18	9586	8973	8989	8973
q19	1827	1788	1769	1769
q20	2378	2032	2069	2032
q21	8826	8466	8384	8384
q22	885	799	792	792
Total cold run time: 81380 ms
Total hot run time: 77679 ms

doris-robot · 2023-11-21T11:06:53Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.89 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17099339088 Bytes

bobhan1 · 2023-11-21T12:06:36Z

run p0

zhannngchen

LGTM

github-actions · 2023-11-21T13:02:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-11-21T13:02:14Z

PR approved by anyone and no changes requested.

yiguolei

LGTM

liaoxin01

LGTM

…27188)" This reverts commit 00c8bab.

…27188)" (#28096) This reverts commit 00c8bab.

)

…pache#27188)" (apache#28096) This reverts commit 00c8bab.

) fix#27188, #28096

…che#28105) fixapache#27188, apache#28096

bobhan1 force-pushed the make-merge-on-write-by-default branch 2 times, most recently from f5a014f to e2f465bCompare November 17, 2023 07:58

bobhan1 force-pushed the make-merge-on-write-by-default branch from 212c032 to f825794Compare November 17, 2023 10:24

bobhan1 force-pushed the make-merge-on-write-by-default branch from 299c54a to 56161b8Compare November 17, 2023 11:23

bobhan1 force-pushed the make-merge-on-write-by-default branch from 56161b8 to c0fab0dCompare November 17, 2023 11:25

bobhan1 marked this pull request as ready for review November 20, 2023 02:40

bobhan1 force-pushed the make-merge-on-write-by-default branch from c0fab0d to db91e48Compare November 20, 2023 02:49

bobhan1 added 7 commits November 21, 2023 14:45

update
8bd9712

update
948fb3f

update
471e03e

update
4498b66

update
492cdbf

update
d955008

add docs
1bfadb1

bobhan1 force-pushed the make-merge-on-write-by-default branch from db91e48 to 1bfadb1Compare November 21, 2023 07:25

fix
fc3adfa

zhannngchen reviewed Nov 21, 2023
View reviewed changes

Comment threaddocs/zh-CN/docs/data-table/data-model.md

zhannngchen approved these changes Nov 21, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2023

github-actionsBot added the reviewed label Nov 21, 2023

yiguolei approved these changes Dec 6, 2023
View reviewed changes

liaoxin01 approved these changes Dec 6, 2023
View reviewed changes

yiguolei merged commit 00c8bab into apache:masterDec 6, 2023

yiguolei added the kind/behavior-changed label Dec 6, 2023

yiguolei added a commit that referenced this pull request Dec 7, 2023
Revert "[feature](merge-on-write) enable merge-on-write by default (#…
a79a127
…27188)" This reverts commit 00c8bab.

yiguolei mentioned this pull request Dec 7, 2023
Revert "[feature](merge-on-write) enable merge-on-write by default" #28096
Merged

zhannngchen pushed a commit that referenced this pull request Dec 7, 2023
Revert "[feature](merge-on-write) enable merge-on-write by default (#…
8c79b86
…27188)" (#28096) This reverts commit 00c8bab.

bobhan1 mentioned this pull request Dec 7, 2023
[feature](merge-on-write) enable merge-on-write by default again #28105
Merged

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
[feature](merge-on-write) enable merge-on-write by default (apache#27188
c38ce57
)

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Revert "[feature](merge-on-write) enable merge-on-write by default (a…
95100b5
…pache#27188)" (apache#28096) This reverts commit 00c8bab.

zhannngchen pushed a commit that referenced this pull request Dec 14, 2023
[feature](merge-on-write) enable merge-on-write by default again (#28105
4300fdc
) fix#27188, #28096

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
[feature](merge-on-write) enable merge-on-write by default again (apa…
653aef8
…che#28105) fixapache#27188, apache#28096

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[feature](merge-on-write) enable merge-on-write by default again (apa…
259e7d2
…che#28105) fixapache#27188, apache#28096

Uh oh!

Conversation

bobhan1 commented Nov 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

zhannngchen commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

bobhan1 commented Nov 20, 2023

Uh oh!

doris-robot commented Nov 20, 2023

Uh oh!

doris-robot commented Nov 20, 2023

Uh oh!

bobhan1 commented Nov 21, 2023

Uh oh!

doris-robot commented Nov 21, 2023

Uh oh!

doris-robot commented Nov 21, 2023

Uh oh!

bobhan1 commented Nov 21, 2023

Uh oh!

doris-robot commented Nov 21, 2023

Uh oh!

doris-robot commented Nov 21, 2023

Uh oh!

bobhan1 commented Nov 21, 2023

Uh oh!

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 21, 2023

Uh oh!

github-actionsBot commented Nov 21, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bobhan1 commented Nov 17, 2023•
edited
Loading