[fix](move-memtable) ignore single replica load when move memtable by kaijchen · Pull Request #32845 · apache/doris

kaijchen · 2024-03-26T09:59:17Z

Proposed changes

Single replica load is not compatible with memtable on sink node.
Enabling single replica load when move memtable will cause missing location in OlapTableSink plan.

Ignore single replica load config when memtable on sink node is enabled.
Namely enable_single_replica_load config in fe.conf and enable_single_replica_insert in session variable.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…n sink node

doris-robot · 2024-03-26T09:59:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

kaijchen · 2024-03-26T10:07:12Z

run buildall

kaijchen · 2024-03-26T12:30:58Z

run buildall

doris-robot · 2024-03-26T12:56:29Z

TPC-H: Total hot run time: 38373 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba, data reload: false
------ Round 1 ----------------------------------
q1	17628	4453	4215	4215
q2	2171	163	151	151
q3	11141	1148	1237	1148
q4	10789	780	780	780
q5	8264	3090	3097	3090
q6	210	130	126	126
q7	1060	627	589	589
q8	9532	2083	2031	2031
q9	7466	6853	6685	6685
q10	8474	3508	3528	3508
q11	440	230	217	217
q12	447	199	199	199
q13	17797	2863	2929	2863
q14	232	203	217	203
q15	516	469	472	469
q16	485	374	372	372
q17	971	635	601	601
q18	7242	6529	6362	6362
q19	1562	1391	1470	1391
q20	536	253	250	250
q21	3584	2908	2825	2825
q22	348	298	304	298
Total cold run time: 110895 ms
Total hot run time: 38373 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4110	4073	4092	4073
q2	327	231	232	231
q3	2987	2878	2856	2856
q4	1897	1578	1614	1578
q5	5290	5374	5375	5374
q6	203	118	119	118
q7	2317	1927	1933	1927
q8	3184	3323	3264	3264
q9	8764	8747	8771	8747
q10	3793	3769	3848	3769
q11	540	446	445	445
q12	716	571	552	552
q13	15854	2854	2845	2845
q14	275	259	272	259
q15	494	463	458	458
q16	471	431	423	423
q17	1731	1511	1469	1469
q18	7418	7294	7131	7131
q19	1647	1558	1528	1528
q20	1891	1732	1720	1720
q21	4936	4658	4777	4658
q22	568	461	433	433
Total cold run time: 69413 ms
Total hot run time: 53858 ms

doris-robot · 2024-03-26T13:08:20Z

TPC-DS: Total hot run time: 181411 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba, data reload: false
query1	935	366	360	360
query2	6556	2045	1741	1741
query3	6710	222	217	217
query4	31732	21260	21361	21260
query5	4277	390	385	385
query6	290	183	180	180
query7	4640	285	288	285
query8	234	170	167	167
query9	9448	2234	2254	2234
query10	555	265	265	265
query11	14701	14320	14215	14215
query12	135	92	90	90
query13	1651	426	421	421
query14	9802	7948	7582	7582
query15	263	197	197	197
query16	8179	263	267	263
query17	1992	577	554	554
query18	2085	281	270	270
query19	347	147	151	147
query20	98	89	88	88
query21	204	123	123	123
query22	5033	4856	4920	4856
query23	33564	32714	32516	32516
query24	12061	2865	2838	2838
query25	628	364	374	364
query26	1804	153	150	150
query27	2967	348	358	348
query28	7744	1848	1841	1841
query29	1007	640	601	601
query30	301	147	144	144
query31	948	749	726	726
query32	87	56	59	56
query33	747	244	244	244
query34	1006	487	474	474
query35	814	638	600	600
query36	1005	916	891	891
query37	265	66	66	66
query38	3577	3426	3418	3418
query39	1464	1455	1421	1421
query40	290	113	114	113
query41	49	48	49	48
query42	108	100	99	99
query43	485	447	462	447
query44	1187	736	756	736
query45	274	267	269	267
query46	1126	695	736	695
query47	1914	1874	1872	1872
query48	464	369	359	359
query49	1222	352	351	351
query50	757	379	377	377
query51	6684	6551	6602	6551
query52	102	97	92	92
query53	346	273	275	273
query54	325	233	242	233
query55	82	87	81	81
query56	247	229	230	229
query57	1214	1149	1140	1140
query58	226	217	210	210
query59	2767	2636	2631	2631
query60	270	249	251	249
query61	114	112	147	112
query62	654	454	451	451
query63	304	274	279	274
query64	6265	4073	4120	4073
query65	3081	3013	3034	3013
query66	1424	385	364	364
query67	15135	14996	15090	14996
query68	6628	532	525	525
query69	605	378	384	378
query70	1233	1199	1159	1159
query71	484	278	271	271
query72	6936	2761	2538	2538
query73	710	319	325	319
query74	8317	6461	6372	6372
query75	3577	2214	2192	2192
query76	4350	874	845	845
query77	695	266	253	253
query78	11083	10155	10119	10119
query79	8402	525	518	518
query80	1429	375	367	367
query81	524	211	216	211
query82	1592	88	82	82
query83	206	149	167	149
query84	285	85	78	78
query85	1499	320	318	318
query86	470	316	298	298
query87	3725	3563	3538	3538
query88	5009	2369	2356	2356
query89	531	366	357	357
query90	2069	174	171	171
query91	174	138	137	137
query92	60	47	47	47
query93	7112	500	479	479
query94	1193	175	173	173
query95	443	325	333	325
query96	628	277	279	277
query97	2690	2499	2462	2462
query98	229	205	212	205
query99	1200	958	973	958
Total cold run time: 311060 ms
Total hot run time: 181411 ms

doris-robot · 2024-03-26T13:11:34Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6f078c7dfe79db7d87e359f816c0e648e93b03ba with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.9 seconds inserted 10000000 Rows, about 478K ops/s

liaoxin01

LGTM

github-actions · 2024-03-27T12:43:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-27T12:43:58Z

PR approved by anyone and no changes requested.

morrySnow · 2024-04-09T09:45:45Z

 } else {
 sink = new OlapTableSink((OlapTable) targetTable, olapTuple, targetPartitionIds,
-analyzer.getContext().getSessionVariable().isEnableSingleReplicaInsert());
+enableSingleReplicaLoad);


do we need add this feature to nereids?

Currently there is no single_replica_load option in nereids.
This PR is for fixing a problem.
If there is a need, we can create a new PR for that feature.

dataroaring

LGTM

liaoxin01 · 2024-04-12T06:12:28Z

run cloud_p1

kaijchen · 2024-04-12T06:45:19Z

run buildall

liaoxin01 · 2024-04-12T14:19:53Z

run fe_ut

liaoxin01 · 2024-04-13T02:01:39Z

run buildall

doris-robot · 2024-04-13T02:19:05Z

TPC-H: Total hot run time: 38555 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
------ Round 1 ----------------------------------
q1	17601	4262	4194	4194
q2	2006	188	194	188
q3	10458	1161	1208	1161
q4	10193	867	801	801
q5	7519	2688	2649	2649
q6	216	132	133	132
q7	1011	607	596	596
q8	9208	2045	2032	2032
q9	8639	6579	6536	6536
q10	8592	3508	3563	3508
q11	462	234	243	234
q12	452	225	230	225
q13	18486	2952	2992	2952
q14	273	231	230	230
q15	523	465	476	465
q16	509	409	398	398
q17	965	756	654	654
q18	7394	6869	6806	6806
q19	8691	1544	1489	1489
q20	1050	325	307	307
q21	3492	2777	2692	2692
q22	358	311	306	306
Total cold run time: 118098 ms
Total hot run time: 38555 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4381	4241	4183	4183
q2	505	264	266	264
q3	3020	2815	2731	2731
q4	1841	1594	1562	1562
q5	5293	5313	5307	5307
q6	210	125	126	125
q7	2252	1899	1875	1875
q8	3209	3331	3306	3306
q9	8632	8575	8915	8575
q10	4085	4040	3919	3919
q11	596	489	497	489
q12	782	645	672	645
q13	17134	3183	3073	3073
q14	311	283	309	283
q15	530	479	465	465
q16	512	455	461	455
q17	1838	1534	1497	1497
q18	8017	7848	7705	7705
q19	1702	1624	1541	1541
q20	1978	1837	1862	1837
q21	5278	5075	4954	4954
q22	559	471	483	471
Total cold run time: 72665 ms
Total hot run time: 55262 ms

doris-robot · 2024-04-13T02:30:54Z

TPC-DS: Total hot run time: 184712 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
query1	895	1120	1118	1118
query2	6169	2845	2494	2494
query3	6639	201	202	201
query4	36901	21645	21457	21457
query5	4141	393	389	389
query6	239	195	179	179
query7	4045	282	285	282
query8	214	166	166	166
query9	5800	2312	2299	2299
query10	369	235	249	235
query11	14511	14326	14176	14176
query12	140	92	87	87
query13	1000	358	357	357
query14	11058	6800	6890	6800
query15	199	181	177	177
query16	6794	255	248	248
query17	1671	570	544	544
query18	1366	293	273	273
query19	198	159	150	150
query20	100	84	89	84
query21	196	133	131	131
query22	5018	4907	4811	4811
query23	33673	33066	33550	33066
query24	11524	3007	3071	3007
query25	544	399	390	390
query26	1074	159	157	157
query27	3285	349	365	349
query28	7108	2077	2110	2077
query29	895	642	628	628
query30	306	172	178	172
query31	944	738	731	731
query32	63	57	55	55
query33	589	255	256	255
query34	935	475	479	475
query35	881	711	707	707
query36	1083	952	950	950
query37	138	69	74	69
query38	3730	3587	3702	3587
query39	1638	1601	1583	1583
query40	174	133	126	126
query41	47	44	42	42
query42	109	102	98	98
query43	613	575	571	571
query44	1425	771	754	754
query45	297	279	281	279
query46	1110	743	750	743
query47	2014	1969	2031	1969
query48	364	321	300	300
query49	798	385	386	385
query50	786	400	421	400
query51	7007	6933	6889	6889
query52	103	93	87	87
query53	345	281	275	275
query54	246	237	214	214
query55	72	72	69	69
query56	240	216	220	216
query57	1199	1111	1132	1111
query58	212	200	198	198
query59	3488	3448	3338	3338
query60	256	231	239	231
query61	94	93	90	90
query62	605	458	461	458
query63	307	277	274	274
query64	4070	4007	3933	3933
query65	3055	3045	3018	3018
query66	766	321	326	321
query67	15778	15278	14816	14816
query68	7117	531	535	531
query69	590	310	309	309
query70	1216	1230	1161	1161
query71	450	280	265	265
query72	6403	2599	2486	2486
query73	819	313	312	312
query74	7074	6526	6399	6399
query75	3250	2388	2327	2327
query76	4211	1088	1139	1088
query77	624	244	247	244
query78	11054	10194	10261	10194
query79	6717	523	521	521
query80	1949	433	419	419
query81	540	249	235	235
query82	1585	98	90	90
query83	347	178	166	166
query84	262	83	85	83
query85	1154	265	260	260
query86	456	289	274	274
query87	3745	3520	3501	3501
query88	6012	2277	2289	2277
query89	506	370	372	370
query90	1850	175	176	175
query91	124	95	93	93
query92	60	48	46	46
query93	5621	512	506	506
query94	1083	182	242	182
query95	376	291	295	291
query96	594	256	256	256
query97	2721	2470	2476	2470
query98	236	216	222	216
query99	1164	844	851	844
Total cold run time: 301563 ms
Total hot run time: 184712 ms

doris-robot · 2024-04-13T02:36:24Z

ClickBench: Total hot run time: 30.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.67	0.06	0.07
query5	0.47	0.48	0.47
query6	1.12	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.53	0.56	0.55
query11	0.16	0.12	0.12
query12	0.14	0.12	0.11
query13	0.61	0.60	0.58
query14	0.75	0.78	0.78
query15	0.83	0.81	0.82
query16	0.36	0.37	0.38
query17	1.02	1.02	1.02
query18	0.19	0.26	0.24
query19	1.84	1.71	1.67
query20	0.02	0.01	0.01
query21	15.41	0.66	0.65
query22	4.86	6.48	2.24
query23	18.33	1.34	1.23
query24	1.72	0.25	0.25
query25	0.15	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.34	0.99	0.97
query29	12.60	3.23	3.24
query30	0.26	0.06	0.06
query31	2.88	0.40	0.38
query32	3.23	0.47	0.46
query33	2.85	2.82	2.88
query34	17.20	4.42	4.41
query35	4.44	4.45	4.46
query36	0.66	0.45	0.46
query37	0.18	0.16	0.15
query38	0.15	0.16	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.69 s
Total hot run time: 30.59 s

doris-robot · 2024-04-13T02:39:28Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bd41d181c19a64bd05c524d0a969eaab59ad6899 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.4 seconds inserted 10000000 Rows, about 746K ops/s

liaoxin01 · 2024-04-16T07:28:45Z

run p0

…32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

…pache#32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

…32845) (#39154) backport #32845 Co-authored-by: Xin Liao <liaoxinbit@126.com>

…pache#32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

kaijchen added 3 commits March 26, 2024 16:53

[fix](stream-load) disable single replica load when enable memtable o…
de9fc12
…n sink node

disable single replica load when move-memtable
3e24b35

add method in session var
ef00b1e

fixup
8a4675e

fix compile
6f078c7

wm1581066 requested review from dataroaring and liaoxin01 March 27, 2024 01:34

liaoxin01 approved these changes Mar 27, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2024

github-actionsBot added the reviewed label Mar 27, 2024

morrySnow reviewed Apr 9, 2024
View reviewed changes

dataroaring approved these changes Apr 9, 2024
View reviewed changes

Merge branch 'master' into fix-srl-move-memtable
d9a3bed

Merge branch 'master' into fix-srl-move-memtable
bd41d18

dataroaring merged commit 9841ac5 into apache:masterApr 17, 2024

kaijchen deleted the fix-srl-move-memtable branch April 17, 2024 02:54

dataroaring pushed a commit that referenced this pull request Apr 24, 2024
[fix](move-memtable) ignore single replica load when move memtable (#…
a229380
…32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

liaoxin01 added the dev/2.1.x label Aug 9, 2024

kaijchen added a commit to kaijchen/doris that referenced this pull request Aug 9, 2024
[fix](move-memtable) ignore single replica load when move memtable (a…
b3675a3
…pache#32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

kaijchen mentioned this pull request Aug 9, 2024
[fix](move-memtable) ignore single replica load when move memtable (#32845) #39154
Merged

dataroaring pushed a commit that referenced this pull request Aug 9, 2024
[fix](move-memtable) ignore single replica load when move memtable (#…
1a3f547
…32845) (#39154) backport #32845 Co-authored-by: Xin Liao <liaoxinbit@126.com>

dataroaring added dev/2.1.6-merged and removed dev/2.1.x labels Aug 9, 2024

cambyzju mentioned this pull request May 16, 2025
[fix](mv) refresh failed while open enable_single_replica_insert #50986
Merged
16 tasks

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](move-memtable) ignore single replica load when move memtable (a…
e9bc5d1
…pache#32845) Co-authored-by: Xin Liao <liaoxinbit@126.com>

Uh oh!

Conversation

kaijchen commented Mar 26, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

kaijchen commented Mar 26, 2024

Uh oh!

kaijchen commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 27, 2024

Uh oh!

github-actionsBot commented Mar 27, 2024

Uh oh!

morrySnowApr 9, 2024

Choose a reason for hiding this comment

Uh oh!

kaijchenApr 17, 2024

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

liaoxin01 commented Apr 12, 2024

Uh oh!

kaijchen commented Apr 12, 2024

Uh oh!

liaoxin01 commented Apr 12, 2024

Uh oh!

liaoxin01 commented Apr 13, 2024

Uh oh!

doris-robot commented Apr 13, 2024

Uh oh!

doris-robot commented Apr 13, 2024

Uh oh!

doris-robot commented Apr 13, 2024

Uh oh!

doris-robot commented Apr 13, 2024

Uh oh!

liaoxin01 commented Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants