[fix](mow) Fix missing mow flag when streamload commit txn from BE introduced by #36237 by gavinchou · Pull Request #36496 · apache/doris

gavinchou · 2024-06-18T15:11:46Z

Fix incomplete optimization introduced by #36237
Opt Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead (#36237)

doris-robot · 2024-06-18T15:11:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

gavinchou · 2024-06-18T15:12:01Z

run buildall

zhannngchen

LGTM

dataroaring

LGTM

github-actions · 2024-06-18T15:20:54Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-18T15:20:56Z

PR approved by anyone and no changes requested.

gavinchou · 2024-06-18T15:25:37Z

run buildall

liaoxin01

LGTM

github-actions · 2024-06-18T15:51:55Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

doris-robot · 2024-06-19T02:31:16Z

TPC-H: Total hot run time: 39507 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4aff9ca34ba67c84920fb0b710421bc47da5216, data reload: false
------ Round 1 ----------------------------------
q1	17630	4384	4301	4301
q2	2033	187	194	187
q3	10441	1200	1109	1109
q4	10195	738	817	738
q5	7452	2627	2609	2609
q6	225	138	130	130
q7	958	606	605	605
q8	9219	2033	2065	2033
q9	8683	6480	6423	6423
q10	8951	3695	3674	3674
q11	455	241	238	238
q12	497	239	222	222
q13	18562	2982	2962	2962
q14	258	222	231	222
q15	524	485	484	484
q16	536	376	370	370
q17	967	731	649	649
q18	8024	7474	7322	7322
q19	7542	1544	1494	1494
q20	660	319	323	319
q21	4799	3079	3840	3079
q22	395	339	337	337
Total cold run time: 119006 ms
Total hot run time: 39507 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4488	4289	4217	4217
q2	386	276	270	270
q3	3075	2930	2902	2902
q4	1902	1750	1735	1735
q5	5487	5469	5519	5469
q6	226	132	128	128
q7	2258	1852	1826	1826
q8	3277	3442	3390	3390
q9	8649	8718	8665	8665
q10	4185	3727	3770	3727
q11	582	492	518	492
q12	790	638	630	630
q13	16239	3204	3159	3159
q14	329	278	288	278
q15	517	495	472	472
q16	494	437	444	437
q17	1819	1499	1510	1499
q18	8192	8008	7739	7739
q19	1837	1642	1550	1550
q20	3039	1866	1840	1840
q21	5115	4866	4571	4571
q22	599	556	549	549
Total cold run time: 73485 ms
Total hot run time: 55545 ms

doris-robot · 2024-06-19T02:42:12Z

TPC-DS: Total hot run time: 174156 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4aff9ca34ba67c84920fb0b710421bc47da5216, data reload: false
query1	927	377	382	377
query2	6465	2405	2316	2316
query3	6646	207	220	207
query4	19427	17352	17256	17256
query5	3644	484	491	484
query6	232	161	158	158
query7	4591	298	287	287
query8	303	295	284	284
query9	8619	2388	2387	2387
query10	561	297	289	289
query11	10595	10068	10065	10065
query12	116	90	91	90
query13	1639	363	370	363
query14	10096	7692	7124	7124
query15	233	186	187	186
query16	7788	256	257	256
query17	1853	551	508	508
query18	1899	265	278	265
query19	205	160	161	160
query20	96	93	85	85
query21	218	131	127	127
query22	4298	4195	4096	4096
query23	34071	33609	33798	33609
query24	11080	2851	2848	2848
query25	599	408	367	367
query26	714	154	151	151
query27	2277	320	332	320
query28	5903	2118	2101	2101
query29	878	674	612	612
query30	237	161	164	161
query31	958	777	747	747
query32	92	55	53	53
query33	748	282	285	282
query34	1053	481	487	481
query35	727	628	625	625
query36	1146	983	984	983
query37	140	86	75	75
query38	2943	2895	2852	2852
query39	885	828	829	828
query40	217	129	129	129
query41	53	53	55	53
query42	125	105	94	94
query43	566	559	535	535
query44	1202	723	728	723
query45	201	158	163	158
query46	1070	743	692	692
query47	1863	1768	1802	1768
query48	365	293	291	291
query49	835	404	404	404
query50	761	386	393	386
query51	6776	6664	6616	6616
query52	106	89	93	89
query53	361	286	291	286
query54	873	442	426	426
query55	78	73	77	73
query56	291	273	284	273
query57	1090	1047	1036	1036
query58	260	257	257	257
query59	3433	3157	3279	3157
query60	294	267	291	267
query61	97	93	93	93
query62	627	435	434	434
query63	311	289	289	289
query64	8511	2247	1872	1872
query65	3176	3097	3109	3097
query66	714	323	329	323
query67	15600	14956	15017	14956
query68	4882	539	540	539
query69	625	452	404	404
query70	1141	1118	1114	1114
query71	409	273	275	273
query72	7857	5642	5731	5642
query73	770	316	319	316
query74	6101	5533	5468	5468
query75	3355	2666	2619	2619
query76	3077	938	973	938
query77	612	290	296	290
query78	10336	9836	9764	9764
query79	2345	506	513	506
query80	1078	468	459	459
query81	593	219	225	219
query82	1090	101	103	101
query83	269	173	175	173
query84	234	92	85	85
query85	1386	334	272	272
query86	474	308	321	308
query87	3274	3126	3102	3102
query88	3715	2310	2308	2308
query89	469	379	373	373
query90	1720	185	181	181
query91	130	99	99	99
query92	56	49	48	48
query93	2340	510	501	501
query94	1148	189	185	185
query95	408	310	316	310
query96	594	270	262	262
query97	3181	3037	3041	3037
query98	226	200	196	196
query99	1169	849	822	822
Total cold run time: 269822 ms
Total hot run time: 174156 ms

doris-robot · 2024-06-19T02:47:42Z

ClickBench: Total hot run time: 31.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4aff9ca34ba67c84920fb0b710421bc47da5216, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.05	0.05
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.50
query6	1.13	0.73	0.72
query7	0.01	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.48	0.50
query10	0.54	0.54	0.55
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.77	0.77
query15	0.82	0.81	0.81
query16	0.36	0.36	0.37
query17	0.98	0.98	0.98
query18	0.23	0.26	0.23
query19	1.91	1.82	1.69
query20	0.02	0.01	0.01
query21	15.45	0.64	0.64
query22	4.59	5.87	2.78
query23	18.31	1.37	1.34
query24	2.11	0.22	0.22
query25	0.15	0.08	0.08
query26	0.28	0.17	0.17
query27	0.09	0.07	0.08
query28	13.24	1.02	0.99
query29	12.62	3.34	3.28
query30	0.26	0.06	0.05
query31	2.87	0.39	0.38
query32	3.26	0.48	0.46
query33	2.89	2.94	2.89
query34	17.25	4.43	4.39
query35	4.50	4.47	4.45
query36	0.64	0.46	0.46
query37	0.18	0.16	0.15
query38	0.14	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.13
query41	0.10	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 31.39 s

zhannngchen

LGTM

Fix incomplete optimization introduced by apache#36237 [Opt](streamload) Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead (apache#36237)

gavinchou · 2024-06-19T19:03:12Z

run buildall

doris-robot · 2024-06-19T19:38:56Z

TPC-H: Total hot run time: 40609 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d20fb928f27aa54a63c4f4bcbb6fbc09433095f, data reload: false
------ Round 1 ----------------------------------
q1	17633	4372	4271	4271
q2	2024	193	196	193
q3	10452	1184	1126	1126
q4	10192	806	853	806
q5	7473	2646	2629	2629
q6	223	133	134	133
q7	951	611	605	605
q8	9221	2050	2099	2050
q9	8996	6495	6494	6494
q10	8985	3763	3767	3763
q11	447	242	237	237
q12	470	230	230	230
q13	17776	2977	2995	2977
q14	283	210	224	210
q15	518	484	481	481
q16	520	378	386	378
q17	968	686	726	686
q18	7963	7476	7373	7373
q19	5766	1501	1416	1416
q20	655	325	338	325
q21	4849	3899	3889	3889
q22	389	341	337	337
Total cold run time: 116754 ms
Total hot run time: 40609 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4395	4270	4281	4270
q2	358	270	266	266
q3	2984	2961	2906	2906
q4	2018	1675	1757	1675
q5	5580	5551	5443	5443
q6	228	127	133	127
q7	2204	1871	1820	1820
q8	3283	3435	3438	3435
q9	8710	8780	8844	8780
q10	4116	3894	3731	3731
q11	595	488	491	488
q12	815	627	635	627
q13	17089	3152	3186	3152
q14	320	264	276	264
q15	531	500	495	495
q16	488	453	429	429
q17	1807	1505	1458	1458
q18	8090	7922	7779	7779
q19	1886	1646	1662	1646
q20	3195	1896	1857	1857
q21	9575	4811	4784	4784
q22	657	542	572	542
Total cold run time: 78924 ms
Total hot run time: 55974 ms

dataroaring

LGTM

github-actions · 2024-06-20T05:17:56Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-06-20T08:07:39Z

TPC-H: Total hot run time: 39968 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d20fb928f27aa54a63c4f4bcbb6fbc09433095f, data reload: false
------ Round 1 ----------------------------------
q1	17921	4459	4287	4287
q2	2031	189	201	189
q3	10525	1120	1098	1098
q4	10229	825	753	753
q5	7487	2676	2678	2676
q6	221	133	145	133
q7	966	601	607	601
q8	9223	2090	2127	2090
q9	8858	6506	6459	6459
q10	8999	3685	3696	3685
q11	458	239	244	239
q12	399	232	223	223
q13	17763	2968	2984	2968
q14	260	225	219	219
q15	514	487	467	467
q16	540	388	383	383
q17	996	749	724	724
q18	8111	7542	7445	7445
q19	7899	1474	1480	1474
q20	680	328	322	322
q21	4869	3190	4014	3190
q22	400	345	343	343
Total cold run time: 119349 ms
Total hot run time: 39968 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4376	4226	4217	4217
q2	389	269	261	261
q3	3187	2865	2960	2865
q4	1990	1720	1752	1720
q5	5570	5488	5493	5488
q6	228	129	141	129
q7	2258	1860	1866	1860
q8	3322	3461	3422	3422
q9	8717	8879	8725	8725
q10	4104	3719	3754	3719
q11	604	493	488	488
q12	847	652	618	618
q13	17563	3121	3174	3121
q14	317	278	281	278
q15	527	486	501	486
q16	504	436	419	419
q17	1807	1544	1486	1486
q18	8158	8042	7912	7912
q19	1833	1610	1654	1610
q20	2124	1892	1840	1840
q21	8768	4750	4877	4750
q22	649	543	536	536
Total cold run time: 77842 ms
Total hot run time: 55950 ms

doris-robot · 2024-06-20T08:18:38Z

TPC-DS: Total hot run time: 174332 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d20fb928f27aa54a63c4f4bcbb6fbc09433095f, data reload: false
query1	924	372	373	372
query2	6481	2544	2393	2393
query3	6627	209	210	209
query4	19236	17363	17433	17363
query5	3731	466	458	458
query6	243	154	154	154
query7	4580	296	288	288
query8	308	309	294	294
query9	8616	2440	2449	2440
query10	569	315	298	298
query11	10520	10019	10138	10019
query12	123	90	83	83
query13	1649	376	366	366
query14	8735	7842	7713	7713
query15	274	192	190	190
query16	7933	254	266	254
query17	1924	533	509	509
query18	1887	265	288	265
query19	199	155	154	154
query20	88	80	82	80
query21	210	124	126	124
query22	4285	4070	3997	3997
query23	33837	33840	33740	33740
query24	10885	2920	2867	2867
query25	620	379	363	363
query26	1279	158	157	157
query27	2963	328	325	325
query28	7442	2121	2101	2101
query29	904	608	618	608
query30	259	151	156	151
query31	981	768	778	768
query32	96	55	55	55
query33	763	297	295	295
query34	1065	488	471	471
query35	775	663	660	660
query36	1126	963	976	963
query37	160	73	69	69
query38	2919	2791	2861	2791
query39	906	862	809	809
query40	206	134	126	126
query41	57	51	53	51
query42	117	109	106	106
query43	593	588	538	538
query44	1175	719	736	719
query45	191	163	157	157
query46	1068	725	715	715
query47	1919	1781	1813	1781
query48	361	294	290	290
query49	859	387	405	387
query50	766	394	389	389
query51	6786	6729	6595	6595
query52	105	93	96	93
query53	360	288	282	282
query54	938	425	445	425
query55	70	74	73	73
query56	277	247	261	247
query57	1131	1069	1029	1029
query58	235	246	243	243
query59	3630	3242	3188	3188
query60	290	273	275	273
query61	90	92	89	89
query62	593	429	439	429
query63	327	286	288	286
query64	9827	2196	1749	1749
query65	3187	3093	3081	3081
query66	1221	319	324	319
query67	15503	14901	14923	14901
query68	4543	532	540	532
query69	617	382	348	348
query70	1165	1124	1143	1124
query71	397	275	271	271
query72	8114	5856	5285	5285
query73	743	325	327	325
query74	5809	5553	5484	5484
query75	3723	2678	2660	2660
query76	2675	972	940	940
query77	470	294	295	294
query78	10306	9871	9814	9814
query79	2466	502	513	502
query80	1026	526	450	450
query81	590	221	219	219
query82	760	103	99	99
query83	266	167	164	164
query84	242	86	83	83
query85	1719	273	269	269
query86	456	292	337	292
query87	3261	3065	3073	3065
query88	4172	2333	2320	2320
query89	477	381	374	374
query90	1808	190	181	181
query91	128	98	99	98
query92	66	52	54	52
query93	1812	496	490	490
query94	1250	191	190	190
query95	407	311	313	311
query96	587	271	266	266
query97	3219	3087	3016	3016
query98	218	201	196	196
query99	1082	856	841	841
Total cold run time: 272578 ms
Total hot run time: 174332 ms

doris-robot · 2024-06-20T08:24:09Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d20fb928f27aa54a63c4f4bcbb6fbc09433095f, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.49	0.49
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.49	0.49
query10	0.56	0.56	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.60	0.59	0.60
query14	0.78	0.78	0.77
query15	0.84	0.80	0.80
query16	0.36	0.35	0.36
query17	1.02	0.97	0.98
query18	0.22	0.25	0.23
query19	1.75	1.77	1.69
query20	0.01	0.01	0.01
query21	15.41	0.65	0.64
query22	4.34	6.41	2.37
query23	18.24	1.36	1.37
query24	2.19	0.22	0.22
query25	0.17	0.09	0.08
query26	0.27	0.18	0.17
query27	0.07	0.08	0.07
query28	13.20	1.01	1.01
query29	12.59	3.31	3.28
query30	0.26	0.07	0.06
query31	2.84	0.39	0.38
query32	3.26	0.48	0.47
query33	2.88	2.91	2.95
query34	17.17	4.41	4.38
query35	4.47	4.50	4.49
query36	0.65	0.48	0.46
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.17	0.18
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.46 s
Total hot run time: 31.11 s

…troduced by #36237 (#36496) Fix incomplete optimization introduced by #36237 [Opt](streamload) Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead (#36237)

…troduced by apache#36237 (apache#36496) Fix incomplete optimization introduced by apache#36237 [Opt](streamload) Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead (apache#36237)

zhannngchen previously approved these changes Jun 18, 2024
View reviewed changes

dataroaring previously approved these changes Jun 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

github-actionsBot added the reviewed label Jun 18, 2024

gavinchou dismissed stale reviews from dataroaring and zhannngchen via f4aff9cJune 18, 2024 15:25

gavinchou force-pushed the gavin-fix-mow-sc branch from 4757977 to f4aff9cCompare June 18, 2024 15:25

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

liaoxin01 previously approved these changes Jun 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

dataroaring previously approved these changes Jun 19, 2024
View reviewed changes

zhannngchen previously approved these changes Jun 19, 2024
View reviewed changes

[fix](mow) Fix missing mow flag when generating plan for streamload
0d20fb9
Fix incomplete optimization introduced by apache#36237 [Opt](streamload) Commit txn on BE coordinator in cloud mode to avoid RPC and lock overhead (apache#36237)

gavinchou dismissed stale reviews from zhannngchen, dataroaring, and liaoxin01 via 0d20fb9June 19, 2024 16:23

gavinchou force-pushed the gavin-fix-mow-sc branch from f4aff9c to 0d20fb9Compare June 19, 2024 16:23

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jun 19, 2024

dataroaring approved these changes Jun 20, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024

Yukang-Lian approved these changes Jun 20, 2024
View reviewed changes

gavinchou merged commit 294437c into apache:masterJun 20, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Uh oh!

Conversation

gavinchou commented Jun 18, 2024

Uh oh!

doris-robot commented Jun 18, 2024

Uh oh!

gavinchou commented Jun 18, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 18, 2024

Uh oh!

github-actionsBot commented Jun 18, 2024

Uh oh!

gavinchou commented Jun 18, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 18, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

gavinchou commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

doris-robot commented Jun 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants